Sep 15 21:23:53.991678 ip-10-0-130-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 15 21:23:53.991789 ip-10-0-130-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 15 21:23:53.991806 ip-10-0-130-52 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 15 21:23:53.992003 ip-10-0-130-52 systemd[1]: Failed to start Kubernetes Kubelet. Sep 15 21:24:04.007774 ip-10-0-130-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 15 21:24:04.007793 ip-10-0-130-52 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9554f49d6b424ea0a9b22aa2f97024f3 -- Sep 15 21:25:50.553405 ip-10-0-130-52 systemd[1]: Starting Kubernetes Kubelet... Sep 15 21:25:50.920709 ip-10-0-130-52 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:25:50.920709 ip-10-0-130-52 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 21:25:50.920709 ip-10-0-130-52 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:25:50.920709 ip-10-0-130-52 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 21:25:50.920709 ip-10-0-130-52 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:25:50.923782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.923693 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 21:25:50.927453 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927434 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:25:50.927453 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927449 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:25:50.927453 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927454 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:25:50.927453 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927457 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:25:50.927453 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927460 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:25:50.927453 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927463 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927465 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927468 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927471 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927473 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927477 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927479 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927483 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927485 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927488 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927491 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927497 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927501 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927505 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927508 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927511 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927514 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927516 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927520 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:25:50.927734 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927522 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927526 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927528 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927531 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927533 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927536 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927538 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927541 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927543 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927546 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927548 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927551 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927553 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927555 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927558 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927561 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927564 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927567 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927570 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927572 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:25:50.928271 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927575 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927579 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927582 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927584 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927586 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927589 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927591 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927594 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927596 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927599 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927601 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927604 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927607 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927609 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927612 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927614 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927617 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927619 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927622 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927624 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:25:50.928806 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927627 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927629 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927632 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927634 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927637 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927639 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927643 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927646 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927649 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927651 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927654 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927659 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927662 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927666 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927669 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927674 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927677 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927680 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927682 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:25:50.929315 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927685 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927687 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927690 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927693 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927695 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927698 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927700 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927703 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927705 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927707 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927710 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927712 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927715 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927717 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927720 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.927723 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928415 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928421 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928424 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928427 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:25:50.929775 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928430 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928433 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928436 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928439 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928442 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928444 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928446 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928451 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928455 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928458 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928460 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928463 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928465 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928468 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928471 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928473 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928476 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928478 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928480 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928483 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:25:50.930276 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928486 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928488 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928490 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928493 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928495 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928498 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928500 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928503 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928506 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928508 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928514 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928516 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928519 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928522 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928525 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928529 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928533 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928536 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928538 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928541 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:25:50.930771 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928544 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928546 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928549 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928551 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928554 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928557 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928559 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928562 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928564 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928567 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928569 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928572 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928574 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928576 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928579 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928581 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928584 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928586 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928589 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928592 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:25:50.931280 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928594 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928597 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928601 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928603 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928606 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928609 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928611 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928614 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928617 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928619 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928622 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928625 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928627 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928630 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928633 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928635 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928637 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928640 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928642 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928644 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:25:50.931783 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928647 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928649 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928651 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928654 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928656 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928659 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928661 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928663 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928666 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928668 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928670 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928673 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928675 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928679 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.928683 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929055 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929060 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929063 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929066 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:25:50.932296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929069 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929073 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929076 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929079 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929082 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929084 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929087 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929089 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929092 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929111 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929114 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929117 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929119 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929122 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929124 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929127 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929129 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929132 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929134 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:25:50.932768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929137 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929139 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929142 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929144 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929147 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929149 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929152 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929155 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929161 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929164 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929167 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929170 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929174 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929176 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929180 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929184 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929187 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929190 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929192 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:25:50.933296 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929195 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929197 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929199 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929202 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929204 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929206 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929209 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929211 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929213 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929216 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929218 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929220 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929223 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929225 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929228 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929230 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929232 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929236 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929239 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929242 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:25:50.933781 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929245 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929248 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929251 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929255 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929258 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929260 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929263 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929265 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929268 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929270 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929273 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929276 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929278 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929281 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929283 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929285 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929288 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929290 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929292 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929296 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:25:50.934275 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929298 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929300 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929303 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929305 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929308 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929311 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929313 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929316 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929318 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929321 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929323 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929326 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929328 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929332 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929334 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929338 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.929340 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929416 2575 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929450 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929456 2575 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929460 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 21:25:50.934790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929465 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929468 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929472 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929476 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929479 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929482 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929485 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929489 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929492 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929495 2575 flags.go:64] FLAG: --cgroup-root="" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929498 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929501 2575 flags.go:64] FLAG: --client-ca-file="" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929504 2575 flags.go:64] FLAG: --cloud-provider="external" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929506 2575 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929511 2575 flags.go:64] FLAG: --cluster-domain="" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929513 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929517 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929519 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929523 2575 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929526 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929529 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929532 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929535 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929538 2575 flags.go:64] FLAG: --contention-profiling="false" Sep 15 21:25:50.935309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929542 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929545 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929549 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929552 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929556 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929559 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929562 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929565 2575 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929567 2575 flags.go:64] FLAG: --enable-server="true" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929571 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929575 2575 flags.go:64] FLAG: --event-burst="100" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929578 2575 flags.go:64] FLAG: --event-qps="50" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929581 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929583 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929586 2575 flags.go:64] FLAG: --eviction-hard="" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929589 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929592 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929595 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929598 2575 flags.go:64] FLAG: --eviction-soft="" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929600 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929603 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929606 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929609 2575 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929612 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929615 2575 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 21:25:50.935880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929617 2575 flags.go:64] FLAG: --feature-gates="" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929621 2575 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929624 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929627 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929630 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929633 2575 flags.go:64] FLAG: --healthz-port="10248" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929635 2575 flags.go:64] FLAG: --help="false" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929639 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-130-52.ec2.internal" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929642 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929646 2575 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929649 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929652 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929655 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929658 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929661 2575 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929664 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929666 2575 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929670 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929673 2575 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929676 2575 flags.go:64] FLAG: --kube-reserved="" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929679 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929681 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929684 2575 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929687 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 21:25:50.936525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929689 2575 flags.go:64] FLAG: --lock-file="" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929692 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929695 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929697 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929703 2575 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929706 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929708 2575 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929711 2575 flags.go:64] FLAG: --logging-format="text" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929714 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929717 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929719 2575 flags.go:64] FLAG: --manifest-url="" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929722 2575 flags.go:64] FLAG: --manifest-url-header="" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929726 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929737 2575 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929742 2575 flags.go:64] FLAG: --max-pods="110" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929748 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929751 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929755 2575 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929758 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929761 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929764 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929767 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929774 2575 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929777 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929780 2575 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 21:25:50.937143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929783 2575 flags.go:64] FLAG: --pod-cidr="" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929787 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929792 2575 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929796 2575 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929799 2575 flags.go:64] FLAG: --pods-per-core="0" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929801 2575 flags.go:64] FLAG: --port="10250" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929804 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929807 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-066fe10378c465eea" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929810 2575 flags.go:64] FLAG: --qos-reserved="" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929813 2575 flags.go:64] FLAG: --read-only-port="10255" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929816 2575 flags.go:64] FLAG: --register-node="true" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929818 2575 flags.go:64] FLAG: --register-with-taints="" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929822 2575 flags.go:64] FLAG: --registry-burst="10" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929825 2575 flags.go:64] FLAG: --registry-qps="5" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929827 2575 flags.go:64] FLAG: --reserved-cpus="" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929830 2575 flags.go:64] FLAG: --reserved-memory="" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929834 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929837 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929840 2575 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929843 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929846 2575 flags.go:64] FLAG: --runonce="false" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929849 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929852 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929855 2575 flags.go:64] FLAG: --seccomp-default="false" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929858 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 21:25:50.937769 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929862 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929865 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929868 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929871 2575 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929874 2575 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929876 2575 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929879 2575 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929882 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929885 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929888 2575 flags.go:64] FLAG: --system-cgroups="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929891 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929897 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929899 2575 flags.go:64] FLAG: --tls-cert-file="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929902 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929907 2575 flags.go:64] FLAG: --tls-min-version="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929909 2575 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929912 2575 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929915 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929917 2575 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929920 2575 flags.go:64] FLAG: --v="2" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929924 2575 flags.go:64] FLAG: --version="false" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929929 2575 flags.go:64] FLAG: --vmodule="" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929932 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.929935 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 21:25:50.938438 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930047 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930060 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930064 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930067 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930069 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930072 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930075 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930078 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930082 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930085 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930088 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930090 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930093 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930110 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930114 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930116 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930119 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930122 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930125 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930127 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:25:50.939022 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930130 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930133 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930135 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930138 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930140 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930143 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930145 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930148 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930150 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930153 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930156 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930158 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930161 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930164 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930166 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930169 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930172 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930175 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930178 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930180 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:25:50.939540 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930184 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930186 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930189 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930191 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930194 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930197 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930199 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930202 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930204 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930207 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930210 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930213 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930215 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930218 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930221 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930223 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930226 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930228 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930231 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930233 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:25:50.940032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930236 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930239 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930243 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930246 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930249 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930252 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930254 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930256 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930260 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930263 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930266 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930268 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930274 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930276 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930279 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930282 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930284 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930287 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930290 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:25:50.940551 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930292 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930295 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930297 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930300 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930302 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930305 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930308 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930311 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930313 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930316 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930318 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930321 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930323 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930325 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930328 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930330 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930333 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930335 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930338 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:25:50.941016 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.930340 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.930346 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.939190 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.939206 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.939930 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.939940 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940001 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940007 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940010 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940014 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940018 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940022 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940025 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940027 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:25:50.941508 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940030 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940033 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940036 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940039 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940041 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940044 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940047 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940049 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940052 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940055 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940057 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940060 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940063 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940065 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940068 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940071 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940073 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940076 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940078 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940081 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:25:50.941931 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940085 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940088 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940092 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940112 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940116 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940120 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940122 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940125 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940128 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940131 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940133 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940136 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940138 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940141 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940143 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940145 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940148 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940150 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940153 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:25:50.942767 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940155 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940158 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940160 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940162 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940165 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940167 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940170 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940173 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940175 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940178 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940180 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940182 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940185 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940188 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940190 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940194 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940196 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940199 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940202 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940204 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:25:50.943553 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940207 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940209 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940212 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940215 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940217 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940220 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940222 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940225 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940227 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940230 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940232 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940235 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940237 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940239 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940242 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940244 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940247 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940249 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940252 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940254 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:25:50.944054 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940256 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940259 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940262 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940264 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940267 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940271 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940274 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940277 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940280 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940282 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940285 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940288 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.940293 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940408 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:25:50.944606 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940413 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940417 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940420 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940422 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940425 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940428 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940430 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940433 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940435 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940438 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940440 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940443 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940445 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940448 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940450 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940453 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940456 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940458 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940461 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940463 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:25:50.945002 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940466 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940469 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940472 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940474 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940477 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940479 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940482 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940485 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940488 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940491 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940493 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940496 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940498 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940501 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940503 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940506 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940508 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940510 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940513 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940515 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:25:50.945612 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940517 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940520 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940523 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940525 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940527 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940530 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940533 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940535 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940538 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940540 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940542 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940545 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940547 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940550 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940552 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940555 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940557 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940560 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940563 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940566 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:25:50.946173 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940569 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940574 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940577 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940579 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940582 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940584 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940587 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940589 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940592 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940594 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940597 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940601 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940604 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940607 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940610 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940613 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940616 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940619 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940621 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940624 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:25:50.947007 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940627 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940630 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940633 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940635 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940638 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940641 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940646 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940648 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940651 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940654 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940656 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940659 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940662 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940667 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940671 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940673 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940676 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:25:50.947745 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:50.940678 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.940683 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.940806 2575 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.944380 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.945280 2575 server.go:1019] "Starting client certificate rotation" Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.945393 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.945436 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 21:25:50.948238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.947519 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 21:25:50.948701 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.948681 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 21:25:50.969212 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.969191 2575 log.go:25] "Validated CRI v1 runtime API" Sep 15 21:25:50.972683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.972661 2575 log.go:25] "Validated CRI v1 image API" Sep 15 21:25:50.973613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.973595 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 21:25:50.976921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.976901 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 21:25:50.979032 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.979008 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ac945562-a588-40b9-bd3b-f1560c419407:/dev/nvme0n1p4 e26facce-2553-45c8-8f11-79ee58b71d66:/dev/nvme0n1p3] Sep 15 21:25:50.979141 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.979030 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 21:25:50.985485 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.985363 2575 manager.go:216] Machine: {Timestamp:2026-09-15 21:25:50.983742147 +0000 UTC m=+0.323221610 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3071863 MemoryCapacity:32812142592 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f82a4c1275501303a9afb8c9e7eb8 SystemUUID:ec2f82a4-c127-5501-303a-9afb8c9e7eb8 BootID:9554f49d-6b42-4ea0-a9b2-2aa2f97024f3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406069248 Type:vfs Inodes:4005388 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:c7:55:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:c7:55:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:4d:0a:54:3a:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812142592 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 21:25:50.985485 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.985472 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 21:25:50.985637 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.985557 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 21:25:50.988428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.988403 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 21:25:50.988564 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.988431 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 21:25:50.989149 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.989138 2575 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 21:25:50.989186 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.989150 2575 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 21:25:50.989186 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.989159 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 21:25:50.989802 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.989791 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 21:25:50.990873 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.990864 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 21:25:50.994436 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994425 2575 state_mem.go:36] "Initialized new in-memory state store" Sep 15 21:25:50.994549 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994539 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 21:25:50.994578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994569 2575 kubelet.go:486] "Attempting to sync node with API server" Sep 15 21:25:50.994609 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994586 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 21:25:50.994609 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994604 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 21:25:50.994662 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994614 2575 kubelet.go:398] "Adding apiserver pod source" Sep 15 21:25:50.994662 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.994622 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 21:25:50.995034 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.995021 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 21:25:50.995118 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.995047 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 21:25:50.997150 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.997131 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 21:25:50.997550 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.997533 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-js957" Sep 15 21:25:50.999603 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.999590 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 21:25:50.999650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:50.999622 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 21:25:51.001325 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001313 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 21:25:51.001361 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001331 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 21:25:51.001361 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001337 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 21:25:51.001361 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001357 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001366 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001371 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001384 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001389 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001395 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001400 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001412 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 21:25:51.001446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.001417 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 21:25:51.002128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.002118 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 21:25:51.002160 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.002129 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 21:25:51.003155 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.003132 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 21:25:51.003155 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.003139 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 21:25:51.004806 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.004792 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 21:25:51.006018 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.006005 2575 server.go:1268] "Started kubelet" Sep 15 21:25:51.006649 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.006610 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 21:25:51.006786 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.006580 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 21:25:51.006935 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.006901 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 21:25:51.007026 ip-10-0-130-52 systemd[1]: Started Kubernetes Kubelet. Sep 15 21:25:51.008441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.008396 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 21:25:51.008805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.008788 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-js957" Sep 15 21:25:51.008935 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.008885 2575 server.go:310] "Adding debug handlers to kubelet server" Sep 15 21:25:51.010692 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.010673 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 21:25:51.010775 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.009806 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.ec2.internal.18d59ba2ec971f68 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.ec2.internal,UID:ip-10-0-130-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.ec2.internal,},FirstTimestamp:2026-09-15 21:25:51.005974376 +0000 UTC m=+0.345453817,LastTimestamp:2026-09-15 21:25:51.005974376 +0000 UTC m=+0.345453817,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.ec2.internal,}" Sep 15 21:25:51.015601 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.015582 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 21:25:51.016270 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.016256 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 21:25:51.016425 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.016409 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.016996 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.016980 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 21:25:51.016996 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.016983 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 21:25:51.017150 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017007 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 21:25:51.017150 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017136 2575 reconstruct.go:97] "Volume reconstruction finished" Sep 15 21:25:51.017150 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017146 2575 reconciler.go:29] "Reconciler: start to sync state" Sep 15 21:25:51.017581 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017565 2575 factory.go:55] Registering systemd factory Sep 15 21:25:51.017626 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017602 2575 factory.go:223] Registration of the systemd container factory successfully Sep 15 21:25:51.017856 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017842 2575 factory.go:153] Registering CRI-O factory Sep 15 21:25:51.017913 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017859 2575 factory.go:223] Registration of the crio container factory successfully Sep 15 21:25:51.017972 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017910 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 21:25:51.017972 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017929 2575 factory.go:103] Registering Raw factory Sep 15 21:25:51.017972 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.017951 2575 manager.go:1196] Started watching for new ooms in manager Sep 15 21:25:51.018488 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.018475 2575 manager.go:349] Starting recovery of all containers Sep 15 21:25:51.019205 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.019183 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:25:51.025090 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.025060 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-52.ec2.internal\" not found" node="ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.031386 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.031369 2575 manager.go:354] Recovery completed Sep 15 21:25:51.035716 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.035703 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.037401 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.037384 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.037470 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.037418 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.037470 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.037435 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.038052 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038037 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 21:25:51.038124 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038053 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 21:25:51.038124 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038038 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.038124 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038074 2575 state_mem.go:36] "Initialized new in-memory state store" Sep 15 21:25:51.038796 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038779 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.038863 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038811 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.038863 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.038826 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.039357 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.039344 2575 policy_none.go:49] "None policy: Start" Sep 15 21:25:51.039410 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.039361 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 21:25:51.039410 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.039372 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 21:25:51.040413 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.040403 2575 policy_none.go:47] "Start" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.082666 2575 manager.go:337] "Starting Device Plugin manager" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.082707 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.082718 2575 server.go:87] "Starting device plugin registration server" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.083035 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.083053 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.083214 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.083305 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.083315 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.083799 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 21:25:51.085197 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.083850 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.138760 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.138729 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 21:25:51.140247 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.140225 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 21:25:51.140369 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.140272 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 21:25:51.140369 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.140299 2575 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 21:25:51.140369 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.140337 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 21:25:51.142776 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.142756 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:25:51.183740 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.183647 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.184646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.184621 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.184646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.184650 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.184809 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.184659 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.184809 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.184703 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.193057 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.193042 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.193138 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.193064 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-52.ec2.internal\": node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.207252 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.207220 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.240797 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.240765 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal"] Sep 15 21:25:51.240913 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.240837 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.246574 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.246554 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.246659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.246584 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.246659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.246596 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.248023 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248011 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.248194 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248181 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.248242 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248209 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.248908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248893 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.248908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248903 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.248988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248923 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.248988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248935 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.249069 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.248925 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.249069 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.249006 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.250376 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.250363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.250422 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.250392 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:25:51.251123 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.251083 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:25:51.251217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.251129 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:25:51.251217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.251143 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:25:51.272700 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.272682 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.ec2.internal\" not found" node="ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.277133 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.277117 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.ec2.internal\" not found" node="ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.307819 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.307796 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.319143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.319121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e5612082f84072df6a197a760b84512-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.ec2.internal\" (UID: \"0e5612082f84072df6a197a760b84512\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.319254 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.319149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a166729dcef75af09399dee5965f61b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal\" (UID: \"a166729dcef75af09399dee5965f61b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.319254 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.319167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a166729dcef75af09399dee5965f61b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal\" (UID: \"a166729dcef75af09399dee5965f61b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.408949 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.408919 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.420248 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.420220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e5612082f84072df6a197a760b84512-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.ec2.internal\" (UID: \"0e5612082f84072df6a197a760b84512\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.420315 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.420254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a166729dcef75af09399dee5965f61b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal\" (UID: \"a166729dcef75af09399dee5965f61b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.420315 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.420272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a166729dcef75af09399dee5965f61b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal\" (UID: \"a166729dcef75af09399dee5965f61b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.420374 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.420319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a166729dcef75af09399dee5965f61b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal\" (UID: \"a166729dcef75af09399dee5965f61b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.420374 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.420324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e5612082f84072df6a197a760b84512-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.ec2.internal\" (UID: \"0e5612082f84072df6a197a760b84512\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.420374 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.420326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a166729dcef75af09399dee5965f61b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal\" (UID: \"a166729dcef75af09399dee5965f61b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.509715 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.509651 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.576233 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.576193 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.580910 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.580698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:51.610301 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.610257 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.710801 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.710770 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.811391 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.811324 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.911852 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:51.911819 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:51.945319 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.945289 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 21:25:51.945717 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.945439 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 21:25:51.945717 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:51.945473 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 21:25:52.012551 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:52.012520 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:52.013633 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.013597 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:20:51 +0000 UTC" deadline="2028-05-14 01:42:06.503301678 +0000 UTC" Sep 15 21:25:52.013693 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.013634 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14548h16m14.489672256s" Sep 15 21:25:52.112686 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:52.112615 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:52.213292 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:52.213258 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-52.ec2.internal\" not found" Sep 15 21:25:52.227203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.227179 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:25:52.256393 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:52.256356 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda166729dcef75af09399dee5965f61b1.slice/crio-0276e27cb7c57c25a2d507d0115f1ee40e8eb818be92634802932cdbef297500 WatchSource:0}: Error finding container 0276e27cb7c57c25a2d507d0115f1ee40e8eb818be92634802932cdbef297500: Status 404 returned error can't find the container with id 0276e27cb7c57c25a2d507d0115f1ee40e8eb818be92634802932cdbef297500 Sep 15 21:25:52.256592 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:52.256563 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5612082f84072df6a197a760b84512.slice/crio-37bf819d6f5b549a06a2dccb1aa4a6a19c370841240018a80e0773ee741a9e4b WatchSource:0}: Error finding container 37bf819d6f5b549a06a2dccb1aa4a6a19c370841240018a80e0773ee741a9e4b: Status 404 returned error can't find the container with id 37bf819d6f5b549a06a2dccb1aa4a6a19c370841240018a80e0773ee741a9e4b Sep 15 21:25:52.260571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.260555 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:25:52.307234 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.307205 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:25:52.317204 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.317172 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" Sep 15 21:25:52.329985 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.329967 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 21:25:52.330991 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.330976 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" Sep 15 21:25:52.342048 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.342032 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 21:25:52.952919 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.952874 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:25:52.996273 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:52.996242 2575 apiserver.go:52] "Watching apiserver" Sep 15 21:25:53.008635 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.007381 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 21:25:53.008635 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.008044 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-d7wjb","kube-system/konnectivity-agent-qwbjf","kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr","openshift-cluster-node-tuning-operator/tuned-8phkl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal","openshift-multus/multus-additional-cni-plugins-m6mh5","openshift-multus/network-metrics-daemon-lvr9k","openshift-dns/node-resolver-q6js4","openshift-image-registry/node-ca-tmpbr","openshift-multus/multus-w4tb2","openshift-network-diagnostics/network-check-target-4gwdz","openshift-network-operator/iptables-alerter-sbmz4","openshift-ovn-kubernetes/ovnkube-node-278vn"] Sep 15 21:25:53.011260 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.010371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.012967 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.012947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.014217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.013750 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 21:25:53.014217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.013770 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 21:25:53.014217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.013777 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.014217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.014048 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.014217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.014086 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 21:25:53.014571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.014265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.014571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.014351 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nzw9t\"" Sep 15 21:25:53.015710 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.015690 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.016075 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.015916 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.016075 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.016055 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 21:25:53.016778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.016286 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 21:25:53.016778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.016550 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pb8bj\"" Sep 15 21:25:53.017662 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.017642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.018421 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.018399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.018720 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.018692 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.018827 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.018757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 21:25:53.018890 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.018849 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 21:25:53.018954 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.018936 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 21:25:53.019013 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.019001 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.019123 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.019107 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-25bms\"" Sep 15 21:25:53.019352 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.019243 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 21:25:53.020168 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.019980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.020168 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.020094 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:25:53.021827 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.021449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.022057 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.021993 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.022217 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.022200 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrtm\"" Sep 15 21:25:53.022310 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.022246 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.022631 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.022611 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 21:25:53.022730 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.022704 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mb6wk\"" Sep 15 21:25:53.022778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.022617 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.022929 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.022906 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.023537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.023421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.023537 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.023495 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:25:53.024708 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.024404 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.025708 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.025210 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rgs9\"" Sep 15 21:25:53.025708 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.025390 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.028273 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.026397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.028545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.028962 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.029404 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.029536 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tm8hk\"" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.029881 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-device-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4xk\" (UniqueName: \"kubernetes.io/projected/5b0f82ff-57ea-45bb-8de9-a0f05f532866-kube-api-access-dk4xk\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-tuned\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-system-cni-dir\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-var-lib-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-cni-bin\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-ovnkube-config\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-ovnkube-script-lib\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-etc-selinux\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.032670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-sys\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030722 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-tmp\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-cnibin\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9245445d-4b90-4419-a545-329b624eb117-iptables-alerter-script\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-kubelet\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-systemd-units\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-slash\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-systemd\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-cni-netd\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.030978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-registration-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-modprobe-d\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysconfig\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-log-socket\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-socket-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysctl-conf\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.033801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-run\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-var-lib-kubelet\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-node-log\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-etc-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-ovn\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s86\" (UniqueName: \"kubernetes.io/projected/64b5be4d-7fac-400c-a631-d62dd983efc7-kube-api-access-r7s86\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-kubernetes\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysctl-d\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-host\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-env-overrides\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-lib-modules\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkn4\" (UniqueName: \"kubernetes.io/projected/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-kube-api-access-dtkn4\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwls\" (UniqueName: \"kubernetes.io/projected/5ae31410-07a4-43a6-bf46-d40116926d74-kube-api-access-gvwls\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/9245445d-4b90-4419-a545-329b624eb117-kube-api-access-8vhwk\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b5be4d-7fac-400c-a631-d62dd983efc7-ovn-node-metrics-cert\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.034683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.031977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-sys-fs\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-systemd\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032377 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032411 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h5x6x\"" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032508 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.032633 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9245445d-4b90-4419-a545-329b624eb117-host-slash\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.035780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.032744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-run-netns\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.038778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.038751 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 21:25:53.041030 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.040992 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wfls6\"" Sep 15 21:25:53.045176 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.041365 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 21:25:53.048052 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.048030 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fnt4p" Sep 15 21:25:53.060448 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.060426 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fnt4p" Sep 15 21:25:53.118282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.118256 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 21:25:53.133174 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-cni-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-kubelet\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-tmp\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9245445d-4b90-4419-a545-329b624eb117-iptables-alerter-script\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-kubelet-config\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed1d53b4-089f-464e-94dc-20f3887ee86d-host\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-netns\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-daemon-config\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-etc-kubernetes\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysctl-conf\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-var-lib-kubelet\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-systemd-units\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-cni-netd\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-modprobe-d\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-system-cni-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkcls\" (UniqueName: \"kubernetes.io/projected/ad01e04c-9e2d-4081-9f98-018ae6469663-kube-api-access-pkcls\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-log-socket\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-socket-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-run\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-cni-multus\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-ovn\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-kubernetes\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-host\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:53.133805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026a5080-dd49-4fed-8060-427077d7a430-tmp-dir\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-env-overrides\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-lib-modules\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/9245445d-4b90-4419-a545-329b624eb117-kube-api-access-8vhwk\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-sys-fs\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-systemd\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-k8s-cni-cncf-io\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-conf-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.133949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-run-netns\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-device-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4xk\" (UniqueName: \"kubernetes.io/projected/5b0f82ff-57ea-45bb-8de9-a0f05f532866-kube-api-access-dk4xk\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-tuned\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-system-cni-dir\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.134622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-ovnkube-config\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-ovnkube-script-lib\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-dbus\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-cnibin\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-multus-certs\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-etc-selinux\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-sys\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-cnibin\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/026a5080-dd49-4fed-8060-427077d7a430-hosts-file\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-kubelet\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-slash\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-systemd\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-registration-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysconfig\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-cni-bin\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.135499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed1d53b4-089f-464e-94dc-20f3887ee86d-serviceca\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-hostroot\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-node-log\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-etc-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s86\" (UniqueName: \"kubernetes.io/projected/64b5be4d-7fac-400c-a631-d62dd983efc7-kube-api-access-r7s86\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysctl-d\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkn4\" (UniqueName: \"kubernetes.io/projected/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-kube-api-access-dtkn4\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwls\" (UniqueName: \"kubernetes.io/projected/5ae31410-07a4-43a6-bf46-d40116926d74-kube-api-access-gvwls\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b5be4d-7fac-400c-a631-d62dd983efc7-ovn-node-metrics-cert\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9245445d-4b90-4419-a545-329b624eb117-host-slash\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htklf\" (UniqueName: \"kubernetes.io/projected/8ff6f6a8-1a38-427f-938c-36d5df919731-kube-api-access-htklf\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.134984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9-agent-certs\") pod \"konnectivity-agent-qwbjf\" (UID: \"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9\") " pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9-konnectivity-ca\") pod \"konnectivity-agent-qwbjf\" (UID: \"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9\") " pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-socket-dir-parent\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad01e04c-9e2d-4081-9f98-018ae6469663-cni-binary-copy\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.136391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-var-lib-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-cni-bin\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvbh\" (UniqueName: \"kubernetes.io/projected/026a5080-dd49-4fed-8060-427077d7a430-kube-api-access-jfvbh\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvdl\" (UniqueName: \"kubernetes.io/projected/ed1d53b4-089f-464e-94dc-20f3887ee86d-kube-api-access-xqvdl\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.135982 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysctl-conf\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136534 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9245445d-4b90-4419-a545-329b624eb117-iptables-alerter-script\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-systemd-units\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136640 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-var-lib-kubelet\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-cni-netd\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-modprobe-d\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136778 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-sys-fs\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-systemd\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.136907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-log-socket\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137017 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-socket-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.137789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-run\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-ovn\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137136 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.137789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-kubernetes\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-host\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.137789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.138090 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137794 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.138090 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-ovnkube-config\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138090 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.137978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-system-cni-dir\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.138090 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138076 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-lib-modules\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.138323 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138113 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-node-log\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138323 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-slash\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138323 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.138570 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-registration-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.138639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138567 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.138639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-etc-selinux\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.138639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-kubelet\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ae31410-07a4-43a6-bf46-d40116926d74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.138639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-run-netns\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b0f82ff-57ea-45bb-8de9-a0f05f532866-device-dir\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-sys\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138567 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-env-overrides\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ae31410-07a4-43a6-bf46-d40116926d74-cnibin\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysconfig\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-var-lib-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-etc-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9245445d-4b90-4419-a545-329b624eb117-host-slash\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-systemd\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138865 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-host-cni-bin\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.138904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64b5be4d-7fac-400c-a631-d62dd983efc7-ovnkube-script-lib\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.139494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.138959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-sysctl-d\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.139494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.139157 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b5be4d-7fac-400c-a631-d62dd983efc7-run-openvswitch\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.140730 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.140710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-tmp\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.141767 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.141740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-etc-tuned\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.141881 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.141840 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b5be4d-7fac-400c-a631-d62dd983efc7-ovn-node-metrics-cert\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.145883 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.145504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/9245445d-4b90-4419-a545-329b624eb117-kube-api-access-8vhwk\") pod \"iptables-alerter-sbmz4\" (UID: \"9245445d-4b90-4419-a545-329b624eb117\") " pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.146864 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.146756 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" event={"ID":"a166729dcef75af09399dee5965f61b1","Type":"ContainerStarted","Data":"0276e27cb7c57c25a2d507d0115f1ee40e8eb818be92634802932cdbef297500"} Sep 15 21:25:53.147489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.147468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4xk\" (UniqueName: \"kubernetes.io/projected/5b0f82ff-57ea-45bb-8de9-a0f05f532866-kube-api-access-dk4xk\") pod \"aws-ebs-csi-driver-node-95nwr\" (UID: \"5b0f82ff-57ea-45bb-8de9-a0f05f532866\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.148267 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.148218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwls\" (UniqueName: \"kubernetes.io/projected/5ae31410-07a4-43a6-bf46-d40116926d74-kube-api-access-gvwls\") pod \"multus-additional-cni-plugins-m6mh5\" (UID: \"5ae31410-07a4-43a6-bf46-d40116926d74\") " pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.149166 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.149093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s86\" (UniqueName: \"kubernetes.io/projected/64b5be4d-7fac-400c-a631-d62dd983efc7-kube-api-access-r7s86\") pod \"ovnkube-node-278vn\" (UID: \"64b5be4d-7fac-400c-a631-d62dd983efc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.149481 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.149456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkn4\" (UniqueName: \"kubernetes.io/projected/db54dcbf-a0d2-4010-89cd-d0c5bae3af5d-kube-api-access-dtkn4\") pod \"tuned-8phkl\" (UID: \"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d\") " pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.150362 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.150169 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" event={"ID":"0e5612082f84072df6a197a760b84512","Type":"ContainerStarted","Data":"37bf819d6f5b549a06a2dccb1aa4a6a19c370841240018a80e0773ee741a9e4b"} Sep 15 21:25:53.236025 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.235936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed1d53b4-089f-464e-94dc-20f3887ee86d-serviceca\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.236025 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.235981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-hostroot\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236025 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htklf\" (UniqueName: \"kubernetes.io/projected/8ff6f6a8-1a38-427f-938c-36d5df919731-kube-api-access-htklf\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-hostroot\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9-agent-certs\") pod \"konnectivity-agent-qwbjf\" (UID: \"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9\") " pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9-konnectivity-ca\") pod \"konnectivity-agent-qwbjf\" (UID: \"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9\") " pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-socket-dir-parent\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad01e04c-9e2d-4081-9f98-018ae6469663-cni-binary-copy\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvbh\" (UniqueName: \"kubernetes.io/projected/026a5080-dd49-4fed-8060-427077d7a430-kube-api-access-jfvbh\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvdl\" (UniqueName: \"kubernetes.io/projected/ed1d53b4-089f-464e-94dc-20f3887ee86d-kube-api-access-xqvdl\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-cni-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-kubelet\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-kubelet-config\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.236344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed1d53b4-089f-464e-94dc-20f3887ee86d-host\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-netns\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-daemon-config\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-etc-kubernetes\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-socket-dir-parent\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236530 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed1d53b4-089f-464e-94dc-20f3887ee86d-serviceca\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-system-cni-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkcls\" (UniqueName: \"kubernetes.io/projected/ad01e04c-9e2d-4081-9f98-018ae6469663-kube-api-access-pkcls\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-cni-multus\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026a5080-dd49-4fed-8060-427077d7a430-tmp-dir\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-k8s-cni-cncf-io\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-conf-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-dbus\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-cnibin\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-multus-certs\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/026a5080-dd49-4fed-8060-427077d7a430-hosts-file\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.236969 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9-konnectivity-ca\") pod \"konnectivity-agent-qwbjf\" (UID: \"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9\") " pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-cni-bin\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-kubelet-config\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-cni-bin\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.237048 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.237140 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:53.73710562 +0000 UTC m=+3.076585075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-cni-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-kubelet\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-var-lib-cni-multus\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-conf-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-dbus\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.236852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad01e04c-9e2d-4081-9f98-018ae6469663-cni-binary-copy\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-multus-certs\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/026a5080-dd49-4fed-8060-427077d7a430-hosts-file\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026a5080-dd49-4fed-8060-427077d7a430-tmp-dir\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.237761 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.237812 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:25:53.737795664 +0000 UTC m=+3.077275101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:53.237926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-k8s-cni-cncf-io\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.238695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237911 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-etc-kubernetes\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.238695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed1d53b4-089f-464e-94dc-20f3887ee86d-host\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.238695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.237987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-cnibin\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.238695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.238020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-system-cni-dir\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.238695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.238034 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad01e04c-9e2d-4081-9f98-018ae6469663-multus-daemon-config\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.238695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.238046 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad01e04c-9e2d-4081-9f98-018ae6469663-host-run-netns\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.239676 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.239641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9-agent-certs\") pod \"konnectivity-agent-qwbjf\" (UID: \"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9\") " pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.244967 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.244944 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htklf\" (UniqueName: \"kubernetes.io/projected/8ff6f6a8-1a38-427f-938c-36d5df919731-kube-api-access-htklf\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.251340 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.251317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkcls\" (UniqueName: \"kubernetes.io/projected/ad01e04c-9e2d-4081-9f98-018ae6469663-kube-api-access-pkcls\") pod \"multus-w4tb2\" (UID: \"ad01e04c-9e2d-4081-9f98-018ae6469663\") " pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.251523 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.251508 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:25:53.251619 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.251531 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:25:53.251619 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.251546 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:53.251619 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.251617 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:53.751600599 +0000 UTC m=+3.091080054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:53.252516 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.252488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvbh\" (UniqueName: \"kubernetes.io/projected/026a5080-dd49-4fed-8060-427077d7a430-kube-api-access-jfvbh\") pod \"node-resolver-q6js4\" (UID: \"026a5080-dd49-4fed-8060-427077d7a430\") " pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.253086 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.253065 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvdl\" (UniqueName: \"kubernetes.io/projected/ed1d53b4-089f-464e-94dc-20f3887ee86d-kube-api-access-xqvdl\") pod \"node-ca-tmpbr\" (UID: \"ed1d53b4-089f-464e-94dc-20f3887ee86d\") " pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.325612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.325552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" Sep 15 21:25:53.345000 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.344974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sbmz4" Sep 15 21:25:53.355929 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.355905 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:25:53.364905 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.364881 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" Sep 15 21:25:53.372332 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.372310 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:25:53.375441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.375422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8phkl" Sep 15 21:25:53.384360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.384338 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6js4" Sep 15 21:25:53.391364 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.391345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:25:53.401062 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.401042 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tmpbr" Sep 15 21:25:53.406831 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.406815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w4tb2" Sep 15 21:25:53.742044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.742013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:53.742259 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.742071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:53.742259 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.742230 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:53.742259 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.742232 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:53.742400 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.742303 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:54.742282944 +0000 UTC m=+4.081762397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:53.742400 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.742318 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:25:54.742311779 +0000 UTC m=+4.081791209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:53.842436 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:53.842392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:53.842598 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.842568 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:25:53.842598 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.842592 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:25:53.842673 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.842602 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:53.842673 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:53.842660 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:54.842645234 +0000 UTC m=+4.182124669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:54.063725 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:54.063633 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:20:53 +0000 UTC" deadline="2028-05-09 19:31:43.814865253 +0000 UTC" Sep 15 21:25:54.063725 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:54.063671 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14446h5m49.751200192s" Sep 15 21:25:54.363203 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:54.363162 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b5be4d_7fac_400c_a631_d62dd983efc7.slice/crio-ba93c0b02042bfadec7e482566f6bd93a74bdb52472f00f54f4461c22dc8e255 WatchSource:0}: Error finding container ba93c0b02042bfadec7e482566f6bd93a74bdb52472f00f54f4461c22dc8e255: Status 404 returned error can't find the container with id ba93c0b02042bfadec7e482566f6bd93a74bdb52472f00f54f4461c22dc8e255 Sep 15 21:25:54.365700 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:54.365524 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb54dcbf_a0d2_4010_89cd_d0c5bae3af5d.slice/crio-dbbee5413301f580c142faea50e3449d7a5c383dabfc0ff7989d6a6f0f944390 WatchSource:0}: Error finding container dbbee5413301f580c142faea50e3449d7a5c383dabfc0ff7989d6a6f0f944390: Status 404 returned error can't find the container with id dbbee5413301f580c142faea50e3449d7a5c383dabfc0ff7989d6a6f0f944390 Sep 15 21:25:54.371397 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:54.371369 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae31410_07a4_43a6_bf46_d40116926d74.slice/crio-bbdd0654bd89d74efe6857a7d2041ca80839fb13ba8fb9c691d38a4ab962173a WatchSource:0}: Error finding container bbdd0654bd89d74efe6857a7d2041ca80839fb13ba8fb9c691d38a4ab962173a: Status 404 returned error can't find the container with id bbdd0654bd89d74efe6857a7d2041ca80839fb13ba8fb9c691d38a4ab962173a Sep 15 21:25:54.372313 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:54.372282 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1d53b4_089f_464e_94dc_20f3887ee86d.slice/crio-fe97b38e98a4a70473affaeb5cc8acb1bf0871c7d11c0f64b3ea7ab8cac13704 WatchSource:0}: Error finding container fe97b38e98a4a70473affaeb5cc8acb1bf0871c7d11c0f64b3ea7ab8cac13704: Status 404 returned error can't find the container with id fe97b38e98a4a70473affaeb5cc8acb1bf0871c7d11c0f64b3ea7ab8cac13704 Sep 15 21:25:54.373496 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:54.373297 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12fc48c_d6c5_4f9a_9e5b_1c210523b8b9.slice/crio-697cc1b908c93993744ba6c187aecc26792ef741eaa7dce6725fc90abd028b5f WatchSource:0}: Error finding container 697cc1b908c93993744ba6c187aecc26792ef741eaa7dce6725fc90abd028b5f: Status 404 returned error can't find the container with id 697cc1b908c93993744ba6c187aecc26792ef741eaa7dce6725fc90abd028b5f Sep 15 21:25:54.374743 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:25:54.374721 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0f82ff_57ea_45bb_8de9_a0f05f532866.slice/crio-742457f58bd9ec43ad07d0ddb1837c0b4f751d2f6ab3d720a8f62033edd3ba5c WatchSource:0}: Error finding container 742457f58bd9ec43ad07d0ddb1837c0b4f751d2f6ab3d720a8f62033edd3ba5c: Status 404 returned error can't find the container with id 742457f58bd9ec43ad07d0ddb1837c0b4f751d2f6ab3d720a8f62033edd3ba5c Sep 15 21:25:54.750045 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:54.750003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:54.750220 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:54.750075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:54.750220 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.750167 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:54.750220 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.750186 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:54.750335 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.750244 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:56.750228119 +0000 UTC m=+6.089707549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:54.750335 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.750258 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:25:56.750251257 +0000 UTC m=+6.089730687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:54.851422 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:54.851383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:54.851594 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.851561 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:25:54.851594 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.851591 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:25:54.851717 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.851604 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:54.851717 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:54.851675 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:25:56.851654306 +0000 UTC m=+6.191133747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:55.064503 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.064391 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:20:53 +0000 UTC" deadline="2028-04-11 06:51:58.455266679 +0000 UTC" Sep 15 21:25:55.064503 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.064448 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13761h26m3.390822646s" Sep 15 21:25:55.144063 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.144027 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:55.144268 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:55.144205 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:25:55.144647 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.144626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:55.144756 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:55.144734 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:25:55.144825 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.144812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:55.144902 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:55.144883 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:25:55.157195 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.157145 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" event={"ID":"5b0f82ff-57ea-45bb-8de9-a0f05f532866","Type":"ContainerStarted","Data":"742457f58bd9ec43ad07d0ddb1837c0b4f751d2f6ab3d720a8f62033edd3ba5c"} Sep 15 21:25:55.160560 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.160517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwbjf" event={"ID":"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9","Type":"ContainerStarted","Data":"697cc1b908c93993744ba6c187aecc26792ef741eaa7dce6725fc90abd028b5f"} Sep 15 21:25:55.162864 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.162830 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8phkl" event={"ID":"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d","Type":"ContainerStarted","Data":"dbbee5413301f580c142faea50e3449d7a5c383dabfc0ff7989d6a6f0f944390"} Sep 15 21:25:55.166958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.166933 2575 generic.go:356] "Generic (PLEG): container finished" podID="a166729dcef75af09399dee5965f61b1" containerID="f93ea51102bb941f25960c791da3ff3ca5cfa608da000b64bc96deefe4f0bb6f" exitCode=0 Sep 15 21:25:55.167044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.167030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" event={"ID":"a166729dcef75af09399dee5965f61b1","Type":"ContainerDied","Data":"f93ea51102bb941f25960c791da3ff3ca5cfa608da000b64bc96deefe4f0bb6f"} Sep 15 21:25:55.175773 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.175718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" event={"ID":"0e5612082f84072df6a197a760b84512","Type":"ContainerStarted","Data":"1cfe7993c1c16d60833e71ca843898367c28380c8cbaa102f53afe1f4c29d6aa"} Sep 15 21:25:55.181151 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.181100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6js4" event={"ID":"026a5080-dd49-4fed-8060-427077d7a430","Type":"ContainerStarted","Data":"513c793e3a63743c5bf4c4c8c1039cdc0ed7bcde5a80f7a89aea836dd708eea7"} Sep 15 21:25:55.184939 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.184910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tmpbr" event={"ID":"ed1d53b4-089f-464e-94dc-20f3887ee86d","Type":"ContainerStarted","Data":"fe97b38e98a4a70473affaeb5cc8acb1bf0871c7d11c0f64b3ea7ab8cac13704"} Sep 15 21:25:55.186427 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.186372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerStarted","Data":"bbdd0654bd89d74efe6857a7d2041ca80839fb13ba8fb9c691d38a4ab962173a"} Sep 15 21:25:55.190224 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.190100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"ba93c0b02042bfadec7e482566f6bd93a74bdb52472f00f54f4461c22dc8e255"} Sep 15 21:25:55.195168 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.195136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sbmz4" event={"ID":"9245445d-4b90-4419-a545-329b624eb117","Type":"ContainerStarted","Data":"68254ad0590e020e9f79331ad4c0a7b3b3d4fd026b53f8aa69cc62171c8a2e2f"} Sep 15 21:25:55.198088 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:55.198058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w4tb2" event={"ID":"ad01e04c-9e2d-4081-9f98-018ae6469663","Type":"ContainerStarted","Data":"6d26684c41c1937b74a713a97829e288389bbf67a0ba66ede5a86f073c080be2"} Sep 15 21:25:56.219207 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:56.217834 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" event={"ID":"a166729dcef75af09399dee5965f61b1","Type":"ContainerStarted","Data":"6bfa7beee8c94fb7d83b5d3fc68c88f4a325f55b7045daed9ddec293dbb7a4f9"} Sep 15 21:25:56.234525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:56.234462 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.ec2.internal" podStartSLOduration=4.234437907 podStartE2EDuration="4.234437907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:55.200205855 +0000 UTC m=+4.539685307" watchObservedRunningTime="2026-09-15 21:25:56.234437907 +0000 UTC m=+5.573917363" Sep 15 21:25:56.774129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:56.773284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:56.774129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:56.773385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:56.774129 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.773518 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:56.774129 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.773591 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:00.77356867 +0000 UTC m=+10.113048123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:25:56.774129 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.774013 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:56.774129 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.774071 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:26:00.774055667 +0000 UTC m=+10.113535115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:25:56.874401 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:56.874347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:56.874625 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.874541 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:25:56.874625 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.874573 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:25:56.874625 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.874588 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:56.874805 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:56.874667 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:00.874642156 +0000 UTC m=+10.214121609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:25:57.143971 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:57.143801 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:57.143971 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:57.143948 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:25:57.144235 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:57.144217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:57.144387 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:57.144346 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:25:57.144387 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:57.144373 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:57.144522 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:57.144464 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:25:59.145734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:59.145194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:25:59.145734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:59.145218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:25:59.145734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:25:59.145218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:25:59.145734 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:59.145341 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:25:59.145734 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:59.145419 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:25:59.145734 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:25:59.145512 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:00.809421 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:00.809174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:00.809924 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:00.809485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:00.809924 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.809656 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:26:00.809924 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.809724 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:08.809703168 +0000 UTC m=+18.149182602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:26:00.810218 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.810064 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:26:00.810218 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.810108 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:26:08.810096207 +0000 UTC m=+18.149575638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:26:00.909932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:00.909893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:00.910173 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.910088 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:26:00.910173 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.910109 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:26:00.910173 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.910124 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:26:00.910363 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:00.910205 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:08.910184134 +0000 UTC m=+18.249663571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:26:01.141961 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:01.141883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:01.142140 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:01.141984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:01.142140 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:01.142007 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:01.142140 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:01.142071 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:01.142140 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:01.142117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:01.142348 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:01.142255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:03.141459 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:03.141421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:03.141866 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:03.141534 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:03.141866 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:03.141833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:03.141941 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:03.141896 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:03.141941 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:03.141936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:03.142032 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:03.141975 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:05.141578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:05.141537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:05.141578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:05.141574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:05.142054 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:05.141606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:05.142054 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:05.141682 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:05.142168 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:05.142135 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:05.142253 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:05.142233 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:07.141399 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:07.141357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:07.141857 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:07.141407 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:07.141857 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:07.141357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:07.141857 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:07.141505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:07.141857 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:07.141573 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:07.141857 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:07.141677 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:08.876664 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:08.876624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:08.877160 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:08.876688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:08.877160 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.876754 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:26:08.877160 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.876773 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:26:08.877160 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.876819 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:26:24.876803516 +0000 UTC m=+34.216282945 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:26:08.877160 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.876838 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:24.876827408 +0000 UTC m=+34.216306838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:26:08.977745 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:08.977712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:08.977924 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.977859 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:26:08.977924 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.977882 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:26:08.977924 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.977895 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:26:08.978062 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:08.977963 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:24.977943378 +0000 UTC m=+34.317422827 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:26:09.141561 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:09.141493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:09.141686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:09.141493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:09.141686 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:09.141611 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:09.141686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:09.141638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:09.141794 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:09.141766 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:09.141878 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:09.141859 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:11.141569 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:11.141526 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:11.141988 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:11.141651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:11.141988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:11.141658 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:11.141988 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:11.141752 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:11.141988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:11.141680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:11.141988 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:11.141828 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:13.143125 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.141202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:13.143125 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:13.141333 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:13.143125 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.141365 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:13.143125 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:13.141471 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:13.143125 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.141782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:13.143125 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:13.141878 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:13.258277 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.257796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tmpbr" event={"ID":"ed1d53b4-089f-464e-94dc-20f3887ee86d","Type":"ContainerStarted","Data":"1ab00b0c9634fc1e64c4c42bda264ab5e63c3cb3aa01c0bfb82cb4b00d5a822e"} Sep 15 21:26:13.260906 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.260408 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerStarted","Data":"64788dcb052df7b0a2f88368fd1e128d5ae1ca65d1dc959cb13e44732e63ae21"} Sep 15 21:26:13.264084 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.262936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"268c284a2a5c4a16c9db91c467ceff2d85796a62de9e0590c7b447b120457d59"} Sep 15 21:26:13.267695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.267633 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w4tb2" event={"ID":"ad01e04c-9e2d-4081-9f98-018ae6469663","Type":"ContainerStarted","Data":"62901378d7a09edc3f4b9b9cf5e7872faa24368118d6e603685f11f052fdf7d0"} Sep 15 21:26:13.267695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.267669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" event={"ID":"5b0f82ff-57ea-45bb-8de9-a0f05f532866","Type":"ContainerStarted","Data":"9edb015a6f18978f5d641d69667c9d265a596e22a22165e19cee833030040d65"} Sep 15 21:26:13.268426 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.267870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwbjf" event={"ID":"d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9","Type":"ContainerStarted","Data":"dbf24130d739de41cc0e36f0ead82f7f9459a46154dfe42e99adb8b2c519d378"} Sep 15 21:26:13.270747 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.270273 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8phkl" event={"ID":"db54dcbf-a0d2-4010-89cd-d0c5bae3af5d","Type":"ContainerStarted","Data":"cf3b78790feb30a57cd5f0138f7637a08593e79467688866a9e3d1382b66464c"} Sep 15 21:26:13.275563 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.274217 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.ec2.internal" podStartSLOduration=21.27420183 podStartE2EDuration="21.27420183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:56.235834559 +0000 UTC m=+5.575314015" watchObservedRunningTime="2026-09-15 21:26:13.27420183 +0000 UTC m=+22.613681282" Sep 15 21:26:13.275563 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.275161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6js4" event={"ID":"026a5080-dd49-4fed-8060-427077d7a430","Type":"ContainerStarted","Data":"d5d7f2ab978310516414929bc076d42db39b5e928f31fc0570c5458b2021d361"} Sep 15 21:26:13.297665 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.297602 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tmpbr" podStartSLOduration=3.669902709 podStartE2EDuration="22.2975802s" totalImagesPullingTime="18.627677491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:13.275656798 +0000 UTC m=+22.615136249" watchObservedRunningTime="2026-09-15 21:26:13.2975802 +0000 UTC m=+22.637059652" Sep 15 21:26:13.314529 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.314486 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q6js4" podStartSLOduration=3.692935752 podStartE2EDuration="22.31447257s" totalImagesPullingTime="18.621536818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:13.314351263 +0000 UTC m=+22.653830715" watchObservedRunningTime="2026-09-15 21:26:13.31447257 +0000 UTC m=+22.653952021" Sep 15 21:26:13.332032 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.331988 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8phkl" podStartSLOduration=3.670379918 podStartE2EDuration="22.331972187s" totalImagesPullingTime="18.661592269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:13.331779068 +0000 UTC m=+22.671258521" watchObservedRunningTime="2026-09-15 21:26:13.331972187 +0000 UTC m=+22.671451634" Sep 15 21:26:13.345662 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.345613 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qwbjf" podStartSLOduration=7.563313424 podStartE2EDuration="22.345598169s" totalImagesPullingTime="14.782284745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:13.345536874 +0000 UTC m=+22.685016324" watchObservedRunningTime="2026-09-15 21:26:13.345598169 +0000 UTC m=+22.685077618" Sep 15 21:26:13.361401 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.361351 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w4tb2" podStartSLOduration=3.685898036 podStartE2EDuration="22.361332647s" totalImagesPullingTime="18.675434611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:13.36103567 +0000 UTC m=+22.700515121" watchObservedRunningTime="2026-09-15 21:26:13.361332647 +0000 UTC m=+22.700812098" Sep 15 21:26:13.550215 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:13.550128 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:26:14.279000 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.278922 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ae31410-07a4-43a6-bf46-d40116926d74" containerID="64788dcb052df7b0a2f88368fd1e128d5ae1ca65d1dc959cb13e44732e63ae21" exitCode=0 Sep 15 21:26:14.279888 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.279001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerDied","Data":"64788dcb052df7b0a2f88368fd1e128d5ae1ca65d1dc959cb13e44732e63ae21"} Sep 15 21:26:14.282439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.282409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"8581810590ce587e11dfbfbc04b2885f0410360ec7f69c51f54baee1ec668f2e"} Sep 15 21:26:14.282550 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.282491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"889a1ba4968733a135fa7b58e3f1c8a8bdc5555742cce09993212437cf82e6ad"} Sep 15 21:26:14.282550 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.282544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"9e4ea970b15e3178e0dd270e5427445daf51c66133727de1e2dfefcb7dffaa7b"} Sep 15 21:26:14.282633 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.282590 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"b62e14b1652b9489939effe75a4313a66b5b4588f4149b086788972919ab0459"} Sep 15 21:26:14.282669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.282638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"64af17df6104bc6faea2b494cd7341825a1777cbcdf326a6100263556d55c1a1"} Sep 15 21:26:14.701613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:14.701463 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 21:26:15.095385 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.095258 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:26:14.701608715Z","UUID":"eba98387-c6d0-4f4f-9490-12a887fdff8c","Handler":null,"Name":"","Endpoint":""} Sep 15 21:26:15.097045 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.097019 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 21:26:15.097197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.097058 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 21:26:15.140972 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.140941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:15.141191 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.140947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:15.141191 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.141086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:15.141191 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:15.141079 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:15.141362 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:15.141200 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:15.141362 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:15.141275 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:15.287151 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.287115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sbmz4" event={"ID":"9245445d-4b90-4419-a545-329b624eb117","Type":"ContainerStarted","Data":"840d26c3fa8f81edb0ce7aff5f738fa8a403ff796000d5267fe9f07a1c26b744"} Sep 15 21:26:15.289155 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.289130 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" event={"ID":"5b0f82ff-57ea-45bb-8de9-a0f05f532866","Type":"ContainerStarted","Data":"1a1aadc53604cc00d2f424fa371b9b075a2653582d0532e0dce096ffdca928ac"} Sep 15 21:26:15.303609 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:15.303553 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sbmz4" podStartSLOduration=5.676106205 podStartE2EDuration="24.30353174s" totalImagesPullingTime="18.627425535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:15.303218156 +0000 UTC m=+24.642697607" watchObservedRunningTime="2026-09-15 21:26:15.30353174 +0000 UTC m=+24.643011192" Sep 15 21:26:16.298238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:16.298207 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"1d1a57c107f59856029ab542923c20187ad48f1f91fd6d960e520c27e3802a77"} Sep 15 21:26:16.708383 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:16.708288 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:26:16.708995 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:16.708973 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:26:17.141331 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:17.141237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:17.141331 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:17.141244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:17.141537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:17.141369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:17.141537 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:17.141369 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:17.141537 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:17.141474 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:17.141652 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:17.141549 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:17.303498 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:17.303254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" event={"ID":"5b0f82ff-57ea-45bb-8de9-a0f05f532866","Type":"ContainerStarted","Data":"4318c41ed798cac5233075d3865536c090e81c8a09fad41148f409527ef8de9d"} Sep 15 21:26:17.303937 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:17.303836 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwbjf" Sep 15 21:26:17.325669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:17.325612 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95nwr" podStartSLOduration=4.8430051259999995 podStartE2EDuration="26.325594351s" totalImagesPullingTime="21.482589225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:17.325320834 +0000 UTC m=+26.664800286" watchObservedRunningTime="2026-09-15 21:26:17.325594351 +0000 UTC m=+26.665073803" Sep 15 21:26:19.141029 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.140809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:19.141489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.140807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:19.141489 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:19.141065 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:19.141489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.140847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:19.141489 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:19.141165 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:19.141489 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:19.141265 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:19.309711 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.309681 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ae31410-07a4-43a6-bf46-d40116926d74" containerID="697abc8b0d20179f70193672b085a0854fa43540b12c190c4d714f42a9b0dafe" exitCode=0 Sep 15 21:26:19.309869 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.309757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerDied","Data":"697abc8b0d20179f70193672b085a0854fa43540b12c190c4d714f42a9b0dafe"} Sep 15 21:26:19.314005 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.313968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" event={"ID":"64b5be4d-7fac-400c-a631-d62dd983efc7","Type":"ContainerStarted","Data":"ce999d1b8504810b60ac65e297aff8c1b7ebc585774c190f3ef3f659a8254041"} Sep 15 21:26:19.314399 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.314349 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:26:19.314508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.314458 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:26:19.332714 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.332688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:26:19.356361 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:19.356321 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" podStartSLOduration=9.667969746 podStartE2EDuration="28.356306295s" totalImagesPullingTime="18.688336549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:19.354927778 +0000 UTC m=+28.694407219" watchObservedRunningTime="2026-09-15 21:26:19.356306295 +0000 UTC m=+28.695785745" Sep 15 21:26:20.316472 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:20.316444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:26:20.336575 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:20.336542 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:26:21.145927 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.145760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:21.146137 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.145813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:21.146137 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:21.145989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:21.146137 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.145813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:21.146137 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:21.146090 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:21.146284 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:21.146201 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:21.185075 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.185043 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d7wjb"] Sep 15 21:26:21.185715 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.185689 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gwdz"] Sep 15 21:26:21.186931 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.186913 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lvr9k"] Sep 15 21:26:21.320744 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.320649 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ae31410-07a4-43a6-bf46-d40116926d74" containerID="9dc2a8557c5b43a1bcbc2429bc91595bdf15d42a2c8da4e04bea322e161471bd" exitCode=0 Sep 15 21:26:21.321244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.320748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerDied","Data":"9dc2a8557c5b43a1bcbc2429bc91595bdf15d42a2c8da4e04bea322e161471bd"} Sep 15 21:26:21.321244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.320788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:21.321244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.320959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:21.321244 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:21.320968 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:21.321244 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:21.321069 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:21.321244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:21.321208 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:21.321497 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:21.321306 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:22.326979 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:22.326884 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ae31410-07a4-43a6-bf46-d40116926d74" containerID="e8c4c015515080bc7d3b9c415ee935fc5e7de6135fee1dc1bbfb9e90100574af" exitCode=0 Sep 15 21:26:22.326979 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:22.326911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerDied","Data":"e8c4c015515080bc7d3b9c415ee935fc5e7de6135fee1dc1bbfb9e90100574af"} Sep 15 21:26:23.140961 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:23.140924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:23.141162 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:23.140924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:23.141162 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:23.141063 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:23.141162 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:23.140925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:23.141289 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:23.141158 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:23.141289 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:23.141236 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:24.896441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:24.896404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:24.896990 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:24.896462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:24.896990 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.896521 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:26:24.896990 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.896578 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:26:24.896990 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.896596 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs podName:8ff6f6a8-1a38-427f-938c-36d5df919731 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:56.896576302 +0000 UTC m=+66.236055731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs") pod "network-metrics-daemon-lvr9k" (UID: "8ff6f6a8-1a38-427f-938c-36d5df919731") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:26:24.896990 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.896633 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret podName:b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f nodeName:}" failed. No retries permitted until 2026-09-15 21:26:56.896616863 +0000 UTC m=+66.236096295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret") pod "global-pull-secret-syncer-d7wjb" (UID: "b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:26:24.997597 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:24.997561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:24.997782 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.997756 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:26:24.997837 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.997783 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:26:24.997837 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.997798 2575 projected.go:196] Error preparing data for projected volume kube-api-access-4bkhj for pod openshift-network-diagnostics/network-check-target-4gwdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:26:24.997917 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:24.997867 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj podName:4073f3d6-73af-480c-9184-5e6cf6542092 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:56.997849326 +0000 UTC m=+66.337328761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bkhj" (UniqueName: "kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj") pod "network-check-target-4gwdz" (UID: "4073f3d6-73af-480c-9184-5e6cf6542092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:26:25.144496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:25.144463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:25.144692 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:25.144576 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:25.144692 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:25.144583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:25.144692 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:25.144581 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gwdz" podUID="4073f3d6-73af-480c-9184-5e6cf6542092" Sep 15 21:26:25.144908 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:25.144690 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d7wjb" podUID="b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f" Sep 15 21:26:25.144908 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:25.144782 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvr9k" podUID="8ff6f6a8-1a38-427f-938c-36d5df919731" Sep 15 21:26:25.934588 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:25.934488 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-52.ec2.internal" event="NodeReady" Sep 15 21:26:25.935077 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:25.934642 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 21:26:25.976439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:25.976400 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59b5cd6b98-92qmw"] Sep 15 21:26:26.030343 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.030309 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6rdt7"] Sep 15 21:26:26.030510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.030385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.034288 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.034261 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 21:26:26.034439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.034388 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 21:26:26.034548 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.034532 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jjzfj\"" Sep 15 21:26:26.034630 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.034262 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 21:26:26.040406 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.040290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 21:26:26.045266 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.045236 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sg6pl"] Sep 15 21:26:26.045396 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.045377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.047678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.047659 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 21:26:26.047806 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.047785 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 21:26:26.047859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.047790 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-646rb\"" Sep 15 21:26:26.047859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.047846 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 21:26:26.061566 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.061534 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b5cd6b98-92qmw"] Sep 15 21:26:26.061696 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.061579 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6rdt7"] Sep 15 21:26:26.061696 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.061598 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sg6pl"] Sep 15 21:26:26.061778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.061699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.064951 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.064926 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 21:26:26.065123 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.064962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 21:26:26.065123 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.064929 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gcfd\"" Sep 15 21:26:26.106344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-installation-pull-secrets\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-certificates\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-image-registry-private-configuration\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdl7m\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-kube-api-access-sdl7m\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-trusted-ca\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106715 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/065ab3b7-ad10-488f-9a97-5811583f9d82-ca-trust-extracted\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106715 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-bound-sa-token\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.106715 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.106607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.207859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.207770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-bound-sa-token\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.207859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.207808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.207859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.207863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-installation-pull-secrets\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.207896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.207930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-certificates\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.207961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cdw\" (UniqueName: \"kubernetes.io/projected/a72d9f28-2893-48a1-9089-8dcaf636f332-kube-api-access-m2cdw\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.207996 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.208023 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5cd6b98-92qmw: secret "image-registry-tls" not found Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-image-registry-private-configuration\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdl7m\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-kube-api-access-sdl7m\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.208090 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls podName:065ab3b7-ad10-488f-9a97-5811583f9d82 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:26.708069773 +0000 UTC m=+36.047549204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls") pod "image-registry-59b5cd6b98-92qmw" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82") : secret "image-registry-tls" not found Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-trusted-ca\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d9f28-2893-48a1-9089-8dcaf636f332-config-volume\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.208669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30c0de63-a264-4c60-8166-f0bdbfca7e0c-cert\") pod \"ingress-canary-6rdt7\" (UID: \"30c0de63-a264-4c60-8166-f0bdbfca7e0c\") " pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.208669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72d9f28-2893-48a1-9089-8dcaf636f332-tmp-dir\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.208669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psll7\" (UniqueName: \"kubernetes.io/projected/30c0de63-a264-4c60-8166-f0bdbfca7e0c-kube-api-access-psll7\") pod \"ingress-canary-6rdt7\" (UID: \"30c0de63-a264-4c60-8166-f0bdbfca7e0c\") " pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.208669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/065ab3b7-ad10-488f-9a97-5811583f9d82-ca-trust-extracted\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208607 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/065ab3b7-ad10-488f-9a97-5811583f9d82-ca-trust-extracted\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.208907 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.208690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-certificates\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.209354 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.209311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-trusted-ca\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.213437 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.213417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-image-registry-private-configuration\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.213560 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.213450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-installation-pull-secrets\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.217083 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.217052 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-bound-sa-token\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.217234 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.217209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdl7m\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-kube-api-access-sdl7m\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.309256 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.309218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.309455 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.309289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cdw\" (UniqueName: \"kubernetes.io/projected/a72d9f28-2893-48a1-9089-8dcaf636f332-kube-api-access-m2cdw\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.309455 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.309325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d9f28-2893-48a1-9089-8dcaf636f332-config-volume\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.309455 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.309345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30c0de63-a264-4c60-8166-f0bdbfca7e0c-cert\") pod \"ingress-canary-6rdt7\" (UID: \"30c0de63-a264-4c60-8166-f0bdbfca7e0c\") " pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.309455 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.309367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72d9f28-2893-48a1-9089-8dcaf636f332-tmp-dir\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.309455 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.309387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psll7\" (UniqueName: \"kubernetes.io/projected/30c0de63-a264-4c60-8166-f0bdbfca7e0c-kube-api-access-psll7\") pod \"ingress-canary-6rdt7\" (UID: \"30c0de63-a264-4c60-8166-f0bdbfca7e0c\") " pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.309455 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.309405 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:26:26.309726 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.309489 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls podName:a72d9f28-2893-48a1-9089-8dcaf636f332 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:26.809467491 +0000 UTC m=+36.148946923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls") pod "dns-default-sg6pl" (UID: "a72d9f28-2893-48a1-9089-8dcaf636f332") : secret "dns-default-metrics-tls" not found Sep 15 21:26:26.310321 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.310272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72d9f28-2893-48a1-9089-8dcaf636f332-tmp-dir\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.310603 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.310409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a72d9f28-2893-48a1-9089-8dcaf636f332-config-volume\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.312618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.312595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30c0de63-a264-4c60-8166-f0bdbfca7e0c-cert\") pod \"ingress-canary-6rdt7\" (UID: \"30c0de63-a264-4c60-8166-f0bdbfca7e0c\") " pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.318329 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.318304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cdw\" (UniqueName: \"kubernetes.io/projected/a72d9f28-2893-48a1-9089-8dcaf636f332-kube-api-access-m2cdw\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.318890 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.318865 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psll7\" (UniqueName: \"kubernetes.io/projected/30c0de63-a264-4c60-8166-f0bdbfca7e0c-kube-api-access-psll7\") pod \"ingress-canary-6rdt7\" (UID: \"30c0de63-a264-4c60-8166-f0bdbfca7e0c\") " pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.355507 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.355472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6rdt7" Sep 15 21:26:26.712668 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.712495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:26.712668 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.712660 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:26:26.712668 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.712681 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5cd6b98-92qmw: secret "image-registry-tls" not found Sep 15 21:26:26.712927 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.712742 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls podName:065ab3b7-ad10-488f-9a97-5811583f9d82 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:27.712725254 +0000 UTC m=+37.052204682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls") pod "image-registry-59b5cd6b98-92qmw" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82") : secret "image-registry-tls" not found Sep 15 21:26:26.814036 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.813949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:26.814233 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.814201 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:26:26.814296 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:26.814282 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls podName:a72d9f28-2893-48a1-9089-8dcaf636f332 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:27.814263779 +0000 UTC m=+37.153743211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls") pod "dns-default-sg6pl" (UID: "a72d9f28-2893-48a1-9089-8dcaf636f332") : secret "dns-default-metrics-tls" not found Sep 15 21:26:26.833361 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:26.833328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6rdt7"] Sep 15 21:26:26.837541 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:26.837516 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c0de63_a264_4c60_8166_f0bdbfca7e0c.slice/crio-e048c6c46eaf0ddda50142d7156edb46ad6507827190a8a6568a49ede91b2e1a WatchSource:0}: Error finding container e048c6c46eaf0ddda50142d7156edb46ad6507827190a8a6568a49ede91b2e1a: Status 404 returned error can't find the container with id e048c6c46eaf0ddda50142d7156edb46ad6507827190a8a6568a49ede91b2e1a Sep 15 21:26:27.141017 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.140980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:27.141646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.140977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:27.141646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.140977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:27.143823 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.143801 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 21:26:27.144678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.144385 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 21:26:27.144678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.144490 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjdv\"" Sep 15 21:26:27.144678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.144391 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 21:26:27.146719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.144615 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q9hj2\"" Sep 15 21:26:27.146719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.144905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 21:26:27.340707 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.340623 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ae31410-07a4-43a6-bf46-d40116926d74" containerID="d3279eed5f1647e4a9b8d0cd7c9bf71888d370a74f4b7fedb69df4e26ec6d864" exitCode=0 Sep 15 21:26:27.340878 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.340710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerDied","Data":"d3279eed5f1647e4a9b8d0cd7c9bf71888d370a74f4b7fedb69df4e26ec6d864"} Sep 15 21:26:27.341956 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.341935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6rdt7" event={"ID":"30c0de63-a264-4c60-8166-f0bdbfca7e0c","Type":"ContainerStarted","Data":"e048c6c46eaf0ddda50142d7156edb46ad6507827190a8a6568a49ede91b2e1a"} Sep 15 21:26:27.720452 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.720357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:27.720628 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:27.720529 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:26:27.720628 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:27.720548 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5cd6b98-92qmw: secret "image-registry-tls" not found Sep 15 21:26:27.720628 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:27.720609 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls podName:065ab3b7-ad10-488f-9a97-5811583f9d82 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:29.720588565 +0000 UTC m=+39.060067998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls") pod "image-registry-59b5cd6b98-92qmw" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82") : secret "image-registry-tls" not found Sep 15 21:26:27.820978 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:27.820941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:27.821180 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:27.821087 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:26:27.821180 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:27.821172 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls podName:a72d9f28-2893-48a1-9089-8dcaf636f332 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:29.821157719 +0000 UTC m=+39.160637147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls") pod "dns-default-sg6pl" (UID: "a72d9f28-2893-48a1-9089-8dcaf636f332") : secret "dns-default-metrics-tls" not found Sep 15 21:26:28.350554 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:28.350514 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ae31410-07a4-43a6-bf46-d40116926d74" containerID="721777f8cd32124e71d4ee007214e980b741c085a4390da8ff5b1c1fadfe188c" exitCode=0 Sep 15 21:26:28.351083 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:28.350599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerDied","Data":"721777f8cd32124e71d4ee007214e980b741c085a4390da8ff5b1c1fadfe188c"} Sep 15 21:26:29.357250 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:29.357217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" event={"ID":"5ae31410-07a4-43a6-bf46-d40116926d74","Type":"ContainerStarted","Data":"5b341366220ebf4f05d4e7cc987fa0958fcb454d4c03355af724a27a15c08324"} Sep 15 21:26:29.385229 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:29.385178 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m6mh5" podStartSLOduration=8.831400715000001 podStartE2EDuration="38.385153524s" totalImagesPullingTime="29.553752809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:29.380767183 +0000 UTC m=+38.720246647" watchObservedRunningTime="2026-09-15 21:26:29.385153524 +0000 UTC m=+38.724632976" Sep 15 21:26:29.739359 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:29.739082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:29.739513 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:29.739244 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:26:29.739513 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:29.739385 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5cd6b98-92qmw: secret "image-registry-tls" not found Sep 15 21:26:29.739513 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:29.739460 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls podName:065ab3b7-ad10-488f-9a97-5811583f9d82 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:33.739443378 +0000 UTC m=+43.078922812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls") pod "image-registry-59b5cd6b98-92qmw" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82") : secret "image-registry-tls" not found Sep 15 21:26:29.840162 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:29.840125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:29.840284 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:29.840263 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:26:29.840343 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:29.840334 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls podName:a72d9f28-2893-48a1-9089-8dcaf636f332 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:33.840314656 +0000 UTC m=+43.179794089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls") pod "dns-default-sg6pl" (UID: "a72d9f28-2893-48a1-9089-8dcaf636f332") : secret "dns-default-metrics-tls" not found Sep 15 21:26:30.360927 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:30.360887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6rdt7" event={"ID":"30c0de63-a264-4c60-8166-f0bdbfca7e0c","Type":"ContainerStarted","Data":"e5c0efb7054f5f0e4ce192b5ce5123ad759402af50faca99b3b517e72a5ac03d"} Sep 15 21:26:30.377493 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:30.377440 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6rdt7" podStartSLOduration=2.862743052 podStartE2EDuration="5.377425706s" totalImagesPullingTime="2.514682654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:30.376522536 +0000 UTC m=+39.716001992" watchObservedRunningTime="2026-09-15 21:26:30.377425706 +0000 UTC m=+39.716905157" Sep 15 21:26:31.732203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.732161 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq"] Sep 15 21:26:31.760490 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.760457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq"] Sep 15 21:26:31.760663 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.760586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" Sep 15 21:26:31.763010 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.762987 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 15 21:26:31.763155 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.763030 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8lgd9\"" Sep 15 21:26:31.763933 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.763910 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 15 21:26:31.813991 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.813954 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2npk"] Sep 15 21:26:31.837800 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.837775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:31.841084 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.841060 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 21:26:31.841222 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.841175 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 21:26:31.841569 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.841517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fd7sr\"" Sep 15 21:26:31.841839 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.841817 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 21:26:31.841937 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.841841 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 21:26:31.843180 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.843160 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2npk"] Sep 15 21:26:31.856358 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.856334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msk5m\" (UniqueName: \"kubernetes.io/projected/b5cb82f5-20ca-488d-a342-a13a72016458-kube-api-access-msk5m\") pod \"migrator-8b4d697c5-mxkcq\" (UID: \"b5cb82f5-20ca-488d-a342-a13a72016458\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" Sep 15 21:26:31.957512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.957478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:31.957512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.957516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1014602b-273c-4d80-ab31-7794c4186635-data-volume\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:31.957683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.957534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:31.957683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.957557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msk5m\" (UniqueName: \"kubernetes.io/projected/b5cb82f5-20ca-488d-a342-a13a72016458-kube-api-access-msk5m\") pod \"migrator-8b4d697c5-mxkcq\" (UID: \"b5cb82f5-20ca-488d-a342-a13a72016458\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" Sep 15 21:26:31.957683 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.957649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1014602b-273c-4d80-ab31-7794c4186635-crio-socket\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:31.957774 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.957710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m2s\" (UniqueName: \"kubernetes.io/projected/1014602b-273c-4d80-ab31-7794c4186635-kube-api-access-j7m2s\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:31.966417 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:31.966390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msk5m\" (UniqueName: \"kubernetes.io/projected/b5cb82f5-20ca-488d-a342-a13a72016458-kube-api-access-msk5m\") pod \"migrator-8b4d697c5-mxkcq\" (UID: \"b5cb82f5-20ca-488d-a342-a13a72016458\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" Sep 15 21:26:32.058694 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.058607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m2s\" (UniqueName: \"kubernetes.io/projected/1014602b-273c-4d80-ab31-7794c4186635-kube-api-access-j7m2s\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.058694 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.058677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.058898 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.058704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1014602b-273c-4d80-ab31-7794c4186635-data-volume\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.058898 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.058723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.058898 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.058763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1014602b-273c-4d80-ab31-7794c4186635-crio-socket\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.058898 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.058741 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:26:32.058898 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.058847 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:26:32.058898 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.058887 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:32.558868155 +0000 UTC m=+41.898347589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : configmap "kube-rbac-proxy" not found Sep 15 21:26:32.059152 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.058935 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:32.558906049 +0000 UTC m=+41.898385478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : secret "insights-runtime-extractor-tls" not found Sep 15 21:26:32.059152 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.058981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1014602b-273c-4d80-ab31-7794c4186635-crio-socket\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.059152 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.059124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1014602b-273c-4d80-ab31-7794c4186635-data-volume\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.068905 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.068878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m2s\" (UniqueName: \"kubernetes.io/projected/1014602b-273c-4d80-ab31-7794c4186635-kube-api-access-j7m2s\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.069129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.069115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" Sep 15 21:26:32.349026 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.348997 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq"] Sep 15 21:26:32.352380 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:32.352353 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cb82f5_20ca_488d_a342_a13a72016458.slice/crio-867c2864a6da4c6479c919f5531750d3e5f4ea8cb802e58cbc0a2654965386d1 WatchSource:0}: Error finding container 867c2864a6da4c6479c919f5531750d3e5f4ea8cb802e58cbc0a2654965386d1: Status 404 returned error can't find the container with id 867c2864a6da4c6479c919f5531750d3e5f4ea8cb802e58cbc0a2654965386d1 Sep 15 21:26:32.367496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.367463 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" event={"ID":"b5cb82f5-20ca-488d-a342-a13a72016458","Type":"ContainerStarted","Data":"867c2864a6da4c6479c919f5531750d3e5f4ea8cb802e58cbc0a2654965386d1"} Sep 15 21:26:32.562954 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.562910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.562954 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:32.562960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:32.563232 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.563049 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:26:32.563232 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.563079 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:26:32.563232 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.563145 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:33.563126646 +0000 UTC m=+42.902606092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : configmap "kube-rbac-proxy" not found Sep 15 21:26:32.563232 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:32.563207 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:33.56318769 +0000 UTC m=+42.902667122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : secret "insights-runtime-extractor-tls" not found Sep 15 21:26:33.319671 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:33.319644 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6js4_026a5080-dd49-4fed-8060-427077d7a430/dns-node-resolver/0.log" Sep 15 21:26:33.571382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:33.571284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:33.571382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:33.571340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:33.571614 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.571433 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:26:33.571614 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.571524 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:35.571501384 +0000 UTC m=+44.910980827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : configmap "kube-rbac-proxy" not found Sep 15 21:26:33.571614 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.571523 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:26:33.571614 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.571594 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:35.571578358 +0000 UTC m=+44.911057786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : secret "insights-runtime-extractor-tls" not found Sep 15 21:26:33.773488 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:33.773447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:33.773663 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.773569 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:26:33.773663 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.773581 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5cd6b98-92qmw: secret "image-registry-tls" not found Sep 15 21:26:33.773663 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.773629 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls podName:065ab3b7-ad10-488f-9a97-5811583f9d82 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:41.773617462 +0000 UTC m=+51.113096891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls") pod "image-registry-59b5cd6b98-92qmw" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82") : secret "image-registry-tls" not found Sep 15 21:26:33.874134 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:33.874031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:33.874274 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.874227 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:26:33.874351 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:33.874309 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls podName:a72d9f28-2893-48a1-9089-8dcaf636f332 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:41.874286552 +0000 UTC m=+51.213765986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls") pod "dns-default-sg6pl" (UID: "a72d9f28-2893-48a1-9089-8dcaf636f332") : secret "dns-default-metrics-tls" not found Sep 15 21:26:34.318300 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.318272 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tmpbr_ed1d53b4-089f-464e-94dc-20f3887ee86d/node-ca/0.log" Sep 15 21:26:34.374480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.374445 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" event={"ID":"b5cb82f5-20ca-488d-a342-a13a72016458","Type":"ContainerStarted","Data":"d564878ecf8c0a0e5def603adfa16551ae0221861c90d1b8542d3030a8f9b2fc"} Sep 15 21:26:34.374817 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.374508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" event={"ID":"b5cb82f5-20ca-488d-a342-a13a72016458","Type":"ContainerStarted","Data":"0b1fd12259465d819a897ef4f2c718eb48ca0ca50a402b22fbc6dbf6ed34fdb7"} Sep 15 21:26:34.919216 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.919187 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6rdt7_30c0de63-a264-4c60-8166-f0bdbfca7e0c/serve-healthcheck-canary/0.log" Sep 15 21:26:34.962595 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.962536 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mxkcq" podStartSLOduration=2.29310942 podStartE2EDuration="3.962513612s" totalImagesPullingTime="1.669404192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:34.395881579 +0000 UTC m=+43.735361032" watchObservedRunningTime="2026-09-15 21:26:34.962513612 +0000 UTC m=+44.301993064" Sep 15 21:26:34.963511 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.963475 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-jbkqm"] Sep 15 21:26:34.985614 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.985589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-jbkqm"] Sep 15 21:26:34.985753 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.985680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:34.988489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.988466 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 15 21:26:34.988591 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.988466 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 15 21:26:34.989417 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.989399 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 15 21:26:34.989488 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.989422 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dg9w6\"" Sep 15 21:26:34.989546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:34.989515 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 15 21:26:35.083697 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.083657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-signing-cabundle\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.083847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.083704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-signing-key\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.083847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.083732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsk5\" (UniqueName: \"kubernetes.io/projected/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-kube-api-access-ttsk5\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.184589 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.184510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-signing-cabundle\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.184589 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.184556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-signing-key\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.184589 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.184573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsk5\" (UniqueName: \"kubernetes.io/projected/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-kube-api-access-ttsk5\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.185238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.185214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-signing-cabundle\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.187066 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.187039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-signing-key\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.193385 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.193364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsk5\" (UniqueName: \"kubernetes.io/projected/59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8-kube-api-access-ttsk5\") pod \"service-ca-568869fdc5-jbkqm\" (UID: \"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8\") " pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.294494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.294453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" Sep 15 21:26:35.587873 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.587793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:35.587873 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.587832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:35.588279 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:35.587927 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:26:35.588279 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:35.587949 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:26:35.588279 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:35.587999 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:39.58798617 +0000 UTC m=+48.927465599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : secret "insights-runtime-extractor-tls" not found Sep 15 21:26:35.588279 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:35.588018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:39.588006795 +0000 UTC m=+48.927486225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : configmap "kube-rbac-proxy" not found Sep 15 21:26:35.606333 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:35.606307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-jbkqm"] Sep 15 21:26:35.608985 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:35.608961 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f34210_8fa9_4ada_9b8c_1b1f6b9e6fa8.slice/crio-66d10627c241c1a91a75f3e52da648d17067336fed18f8427a4bcba356936697 WatchSource:0}: Error finding container 66d10627c241c1a91a75f3e52da648d17067336fed18f8427a4bcba356936697: Status 404 returned error can't find the container with id 66d10627c241c1a91a75f3e52da648d17067336fed18f8427a4bcba356936697 Sep 15 21:26:36.382039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:36.381998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" event={"ID":"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8","Type":"ContainerStarted","Data":"66d10627c241c1a91a75f3e52da648d17067336fed18f8427a4bcba356936697"} Sep 15 21:26:38.389159 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:38.389059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" event={"ID":"59f34210-8fa9-4ada-9b8c-1b1f6b9e6fa8","Type":"ContainerStarted","Data":"f61c4425d34c422998853f16ca1e4d650210ba86cc7dbf4c099b73870807441e"} Sep 15 21:26:39.622824 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:39.622773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:39.623304 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:39.622835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:39.623304 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:39.623043 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:26:39.623304 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:39.623144 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls podName:1014602b-273c-4d80-ab31-7794c4186635 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:47.623123873 +0000 UTC m=+56.962603320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2npk" (UID: "1014602b-273c-4d80-ab31-7794c4186635") : secret "insights-runtime-extractor-tls" not found Sep 15 21:26:39.640351 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:39.640276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1014602b-273c-4d80-ab31-7794c4186635-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:41.840568 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:41.840532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:41.841175 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:41.840730 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:26:41.841175 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:41.840753 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5cd6b98-92qmw: secret "image-registry-tls" not found Sep 15 21:26:41.841175 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:41.840834 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls podName:065ab3b7-ad10-488f-9a97-5811583f9d82 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:57.840812652 +0000 UTC m=+67.180292105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls") pod "image-registry-59b5cd6b98-92qmw" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82") : secret "image-registry-tls" not found Sep 15 21:26:41.941498 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:41.941461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:41.941661 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:41.941617 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:26:41.941702 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:41.941682 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls podName:a72d9f28-2893-48a1-9089-8dcaf636f332 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:57.941665163 +0000 UTC m=+67.281144596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls") pod "dns-default-sg6pl" (UID: "a72d9f28-2893-48a1-9089-8dcaf636f332") : secret "dns-default-metrics-tls" not found Sep 15 21:26:47.689192 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:47.689141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:47.691650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:47.691618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1014602b-273c-4d80-ab31-7794c4186635-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2npk\" (UID: \"1014602b-273c-4d80-ab31-7794c4186635\") " pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:47.747739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:47.747701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2npk" Sep 15 21:26:48.018241 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:48.018080 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-jbkqm" podStartSLOduration=11.614550008 podStartE2EDuration="14.018059871s" totalImagesPullingTime="2.403509863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:38.407513601 +0000 UTC m=+47.746993054" watchObservedRunningTime="2026-09-15 21:26:48.018059871 +0000 UTC m=+57.357539322" Sep 15 21:26:48.019343 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:48.019315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2npk"] Sep 15 21:26:48.022560 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:48.022517 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1014602b_273c_4d80_ab31_7794c4186635.slice/crio-b5b2d3c00fa53e233dc2e824c85b6d08e521028441aa7402cb70236febdc8a5e WatchSource:0}: Error finding container b5b2d3c00fa53e233dc2e824c85b6d08e521028441aa7402cb70236febdc8a5e: Status 404 returned error can't find the container with id b5b2d3c00fa53e233dc2e824c85b6d08e521028441aa7402cb70236febdc8a5e Sep 15 21:26:48.419931 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:48.419895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2npk" event={"ID":"1014602b-273c-4d80-ab31-7794c4186635","Type":"ContainerStarted","Data":"9acffc3b70e6e8ccb4d6d3e5f8ff410ab2c21fb88236d7148928fee166dc642a"} Sep 15 21:26:48.420084 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:48.419953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2npk" event={"ID":"1014602b-273c-4d80-ab31-7794c4186635","Type":"ContainerStarted","Data":"b5b2d3c00fa53e233dc2e824c85b6d08e521028441aa7402cb70236febdc8a5e"} Sep 15 21:26:50.427117 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:50.427063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2npk" event={"ID":"1014602b-273c-4d80-ab31-7794c4186635","Type":"ContainerStarted","Data":"1cc80e8578677d6475ec1b06c6045605fc63a72c0d54c5291793e92c64b16cee"} Sep 15 21:26:51.431936 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:51.431896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2npk" event={"ID":"1014602b-273c-4d80-ab31-7794c4186635","Type":"ContainerStarted","Data":"ede2f8f14562ad286246501332d6066284fe80029649531bebe0ecdffd935f74"} Sep 15 21:26:51.453610 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:51.453560 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c2npk" podStartSLOduration=17.312971996 podStartE2EDuration="20.453544813s" totalImagesPullingTime="3.140572817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:51.452340735 +0000 UTC m=+60.791820187" watchObservedRunningTime="2026-09-15 21:26:51.453544813 +0000 UTC m=+60.793024264" Sep 15 21:26:52.338376 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:52.338350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-278vn" Sep 15 21:26:56.962146 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:56.962082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:56.962587 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:56.962184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:56.964736 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:56.964712 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 21:26:56.964852 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:56.964741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 21:26:56.975267 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:56.975245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f-original-pull-secret\") pod \"global-pull-secret-syncer-d7wjb\" (UID: \"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f\") " pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:56.975367 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:56.975253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6f6a8-1a38-427f-938c-36d5df919731-metrics-certs\") pod \"network-metrics-daemon-lvr9k\" (UID: \"8ff6f6a8-1a38-427f-938c-36d5df919731\") " pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:57.063063 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.063026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:57.066330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.066306 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 21:26:57.075627 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.075604 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 21:26:57.086806 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.086779 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkhj\" (UniqueName: \"kubernetes.io/projected/4073f3d6-73af-480c-9184-5e6cf6542092-kube-api-access-4bkhj\") pod \"network-check-target-4gwdz\" (UID: \"4073f3d6-73af-480c-9184-5e6cf6542092\") " pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:57.156505 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.156476 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q9hj2\"" Sep 15 21:26:57.160895 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.160880 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjdv\"" Sep 15 21:26:57.164212 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.164199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvr9k" Sep 15 21:26:57.164337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.164199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d7wjb" Sep 15 21:26:57.169379 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.169362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:26:57.335596 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.335561 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp"] Sep 15 21:26:57.357430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.357407 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk"] Sep 15 21:26:57.357582 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.357534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.360761 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.360741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 21:26:57.361288 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.361271 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 15 21:26:57.361392 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.361376 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 21:26:57.361457 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.361401 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 21:26:57.377585 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.377564 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp"] Sep 15 21:26:57.377585 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.377601 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk"] Sep 15 21:26:57.377783 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.377615 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b5cd6b98-92qmw"] Sep 15 21:26:57.377783 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.377693 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.377869 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:26:57.377796 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" podUID="065ab3b7-ad10-488f-9a97-5811583f9d82" Sep 15 21:26:57.380255 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.380223 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 15 21:26:57.380381 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.380314 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 15 21:26:57.380381 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.380343 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 15 21:26:57.380381 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.380364 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 15 21:26:57.424760 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.424730 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757c5b4b9d-7pz5c"] Sep 15 21:26:57.439438 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.439411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.447734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.447708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757c5b4b9d-7pz5c"] Sep 15 21:26:57.451678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.451649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:57.466944 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.466915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-ca\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.467141 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.466949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.467141 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.466989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89427796-61c1-4092-a49a-042f7dee3a8b-tmp\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.467141 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.467048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89427796-61c1-4092-a49a-042f7dee3a8b-klusterlet-config\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.467141 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.467089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6m9s\" (UniqueName: \"kubernetes.io/projected/369e5023-7103-4182-866d-2a3dcd0ed422-kube-api-access-n6m9s\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.467314 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.467164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fs4\" (UniqueName: \"kubernetes.io/projected/89427796-61c1-4092-a49a-042f7dee3a8b-kube-api-access-98fs4\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.467314 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.467185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e5023-7103-4182-866d-2a3dcd0ed422-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.467314 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.467200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-hub\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.467314 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.467237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.475125 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.475054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lvr9k"] Sep 15 21:26:57.475516 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.475484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:57.478611 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:57.478576 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff6f6a8_1a38_427f_938c_36d5df919731.slice/crio-f157ab99408c8690ae9d0918bfa99c0f937fb9280692ab503ce4cf1d98b8de21 WatchSource:0}: Error finding container f157ab99408c8690ae9d0918bfa99c0f937fb9280692ab503ce4cf1d98b8de21: Status 404 returned error can't find the container with id f157ab99408c8690ae9d0918bfa99c0f937fb9280692ab503ce4cf1d98b8de21 Sep 15 21:26:57.498493 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.498466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d7wjb"] Sep 15 21:26:57.502215 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:57.502158 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bb12a7_b2b9_4153_b889_21b5cfe1dc1f.slice/crio-54ecf15b0a9b34be6cc39bdd688f2ef85ef8898ef4c5b2e9e00ded91b2f83568 WatchSource:0}: Error finding container 54ecf15b0a9b34be6cc39bdd688f2ef85ef8898ef4c5b2e9e00ded91b2f83568: Status 404 returned error can't find the container with id 54ecf15b0a9b34be6cc39bdd688f2ef85ef8898ef4c5b2e9e00ded91b2f83568 Sep 15 21:26:57.525020 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.524998 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gwdz"] Sep 15 21:26:57.527371 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:57.527344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4073f3d6_73af_480c_9184_5e6cf6542092.slice/crio-e14b921892ce808c3e9036d68fc32e3d01a744b83740521aadbd83b4fbdc3076 WatchSource:0}: Error finding container e14b921892ce808c3e9036d68fc32e3d01a744b83740521aadbd83b4fbdc3076: Status 404 returned error can't find the container with id e14b921892ce808c3e9036d68fc32e3d01a744b83740521aadbd83b4fbdc3076 Sep 15 21:26:57.567737 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.567710 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-bound-sa-token\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.567890 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.567756 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-installation-pull-secrets\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.567890 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.567801 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-certificates\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.568004 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.567890 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdl7m\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-kube-api-access-sdl7m\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.568004 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.567921 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/065ab3b7-ad10-488f-9a97-5811583f9d82-ca-trust-extracted\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.568004 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.567936 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-trusted-ca\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.568203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568006 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-image-registry-private-configuration\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.568203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fs4\" (UniqueName: \"kubernetes.io/projected/89427796-61c1-4092-a49a-042f7dee3a8b-kube-api-access-98fs4\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.568203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e5023-7103-4182-866d-2a3dcd0ed422-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.568203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-hub\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.568393 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568204 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:26:57.568393 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568223 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ab3b7-ad10-488f-9a97-5811583f9d82-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:26:57.568393 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq62b\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-kube-api-access-hq62b\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.568709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568537 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:26:57.568709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.568709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fdb54a01-af83-4d22-9437-a97cb7169382-registry-certificates\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.568709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-ca\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.568709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.568709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-registry-tls\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89427796-61c1-4092-a49a-042f7dee3a8b-tmp\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdb54a01-af83-4d22-9437-a97cb7169382-trusted-ca\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fdb54a01-af83-4d22-9437-a97cb7169382-image-registry-private-configuration\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89427796-61c1-4092-a49a-042f7dee3a8b-klusterlet-config\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6m9s\" (UniqueName: \"kubernetes.io/projected/369e5023-7103-4182-866d-2a3dcd0ed422-kube-api-access-n6m9s\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fdb54a01-af83-4d22-9437-a97cb7169382-ca-trust-extracted\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-bound-sa-token\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fdb54a01-af83-4d22-9437-a97cb7169382-installation-pull-secrets\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e5023-7103-4182-866d-2a3dcd0ed422-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.569006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.568999 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-certificates\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.569502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.569015 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/065ab3b7-ad10-488f-9a97-5811583f9d82-ca-trust-extracted\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.569502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.569064 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/065ab3b7-ad10-488f-9a97-5811583f9d82-trusted-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.570385 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.570354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89427796-61c1-4092-a49a-042f7dee3a8b-tmp\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.570851 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.570822 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-kube-api-access-sdl7m" (OuterVolumeSpecName: "kube-api-access-sdl7m") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "kube-api-access-sdl7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:26:57.570949 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.570858 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:26:57.570995 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.570959 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:26:57.571136 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.571093 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:26:57.571914 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.571882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-ca\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.572140 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.572122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.572590 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.572564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89427796-61c1-4092-a49a-042f7dee3a8b-klusterlet-config\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.572873 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.572858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-hub\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.572916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.572889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369e5023-7103-4182-866d-2a3dcd0ed422-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.578910 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.578885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fs4\" (UniqueName: \"kubernetes.io/projected/89427796-61c1-4092-a49a-042f7dee3a8b-kube-api-access-98fs4\") pod \"klusterlet-addon-workmgr-f9fbcff75-mwsnp\" (UID: \"89427796-61c1-4092-a49a-042f7dee3a8b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.579946 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.579929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6m9s\" (UniqueName: \"kubernetes.io/projected/369e5023-7103-4182-866d-2a3dcd0ed422-kube-api-access-n6m9s\") pod \"cluster-proxy-proxy-agent-84c4cc567b-bnprk\" (UID: \"369e5023-7103-4182-866d-2a3dcd0ed422\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.667132 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.667084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:26:57.670362 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fdb54a01-af83-4d22-9437-a97cb7169382-registry-certificates\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-registry-tls\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdb54a01-af83-4d22-9437-a97cb7169382-trusted-ca\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fdb54a01-af83-4d22-9437-a97cb7169382-image-registry-private-configuration\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670655 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fdb54a01-af83-4d22-9437-a97cb7169382-ca-trust-extracted\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670655 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-bound-sa-token\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670655 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fdb54a01-af83-4d22-9437-a97cb7169382-installation-pull-secrets\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq62b\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-kube-api-access-hq62b\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.670807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670742 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-bound-sa-token\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.670807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670758 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-installation-pull-secrets\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.670807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670773 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdl7m\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-kube-api-access-sdl7m\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.670807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.670789 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/065ab3b7-ad10-488f-9a97-5811583f9d82-image-registry-private-configuration\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:57.671043 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.671015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fdb54a01-af83-4d22-9437-a97cb7169382-ca-trust-extracted\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.671173 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.671148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fdb54a01-af83-4d22-9437-a97cb7169382-registry-certificates\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.671512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.671494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdb54a01-af83-4d22-9437-a97cb7169382-trusted-ca\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.673000 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.672961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fdb54a01-af83-4d22-9437-a97cb7169382-image-registry-private-configuration\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.673492 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.673473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-registry-tls\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.673565 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.673516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fdb54a01-af83-4d22-9437-a97cb7169382-installation-pull-secrets\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.682739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.682718 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-bound-sa-token\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.683429 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.683411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq62b\" (UniqueName: \"kubernetes.io/projected/fdb54a01-af83-4d22-9437-a97cb7169382-kube-api-access-hq62b\") pod \"image-registry-757c5b4b9d-7pz5c\" (UID: \"fdb54a01-af83-4d22-9437-a97cb7169382\") " pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.697579 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.697556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" Sep 15 21:26:57.751307 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.751286 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jjzfj\"" Sep 15 21:26:57.759633 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.759574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:57.872538 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.872491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:57.875281 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.875258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"image-registry-59b5cd6b98-92qmw\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:57.957441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.957403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp"] Sep 15 21:26:57.961561 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:57.961525 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89427796_61c1_4092_a49a_042f7dee3a8b.slice/crio-e42bd2335926602f6ad31b0205e3b6c8a3491ad787f5f5017934253c59bf0502 WatchSource:0}: Error finding container e42bd2335926602f6ad31b0205e3b6c8a3491ad787f5f5017934253c59bf0502: Status 404 returned error can't find the container with id e42bd2335926602f6ad31b0205e3b6c8a3491ad787f5f5017934253c59bf0502 Sep 15 21:26:57.972960 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.972927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") pod \"065ab3b7-ad10-488f-9a97-5811583f9d82\" (UID: \"065ab3b7-ad10-488f-9a97-5811583f9d82\") " Sep 15 21:26:57.973294 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.973082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:57.975653 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.975628 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "065ab3b7-ad10-488f-9a97-5811583f9d82" (UID: "065ab3b7-ad10-488f-9a97-5811583f9d82"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:26:57.975943 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:57.975923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a72d9f28-2893-48a1-9089-8dcaf636f332-metrics-tls\") pod \"dns-default-sg6pl\" (UID: \"a72d9f28-2893-48a1-9089-8dcaf636f332\") " pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:58.007388 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.007364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk"] Sep 15 21:26:58.010558 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:58.010533 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369e5023_7103_4182_866d_2a3dcd0ed422.slice/crio-2fd937fbcc690cb8012980bd687d9d3cdbc526ba9480669612963a1c2f38c2fd WatchSource:0}: Error finding container 2fd937fbcc690cb8012980bd687d9d3cdbc526ba9480669612963a1c2f38c2fd: Status 404 returned error can't find the container with id 2fd937fbcc690cb8012980bd687d9d3cdbc526ba9480669612963a1c2f38c2fd Sep 15 21:26:58.057287 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.057255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757c5b4b9d-7pz5c"] Sep 15 21:26:58.060152 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:58.060120 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb54a01_af83_4d22_9437_a97cb7169382.slice/crio-fabefb12314d4b11852966b9b8cbec74e7d702f33af1e504823836124c406f2f WatchSource:0}: Error finding container fabefb12314d4b11852966b9b8cbec74e7d702f33af1e504823836124c406f2f: Status 404 returned error can't find the container with id fabefb12314d4b11852966b9b8cbec74e7d702f33af1e504823836124c406f2f Sep 15 21:26:58.074072 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.074045 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/065ab3b7-ad10-488f-9a97-5811583f9d82-registry-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:26:58.175286 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.175248 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gcfd\"" Sep 15 21:26:58.183673 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.183436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sg6pl" Sep 15 21:26:58.476480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.476330 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" event={"ID":"369e5023-7103-4182-866d-2a3dcd0ed422","Type":"ContainerStarted","Data":"2fd937fbcc690cb8012980bd687d9d3cdbc526ba9480669612963a1c2f38c2fd"} Sep 15 21:26:58.477937 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.477909 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d7wjb" event={"ID":"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f","Type":"ContainerStarted","Data":"54ecf15b0a9b34be6cc39bdd688f2ef85ef8898ef4c5b2e9e00ded91b2f83568"} Sep 15 21:26:58.481510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.481438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvr9k" event={"ID":"8ff6f6a8-1a38-427f-938c-36d5df919731","Type":"ContainerStarted","Data":"f157ab99408c8690ae9d0918bfa99c0f937fb9280692ab503ce4cf1d98b8de21"} Sep 15 21:26:58.483332 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.483248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" event={"ID":"89427796-61c1-4092-a49a-042f7dee3a8b","Type":"ContainerStarted","Data":"e42bd2335926602f6ad31b0205e3b6c8a3491ad787f5f5017934253c59bf0502"} Sep 15 21:26:58.485040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.484889 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gwdz" event={"ID":"4073f3d6-73af-480c-9184-5e6cf6542092","Type":"ContainerStarted","Data":"e14b921892ce808c3e9036d68fc32e3d01a744b83740521aadbd83b4fbdc3076"} Sep 15 21:26:58.489180 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.487167 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5cd6b98-92qmw" Sep 15 21:26:58.489538 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.489490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" event={"ID":"fdb54a01-af83-4d22-9437-a97cb7169382","Type":"ContainerStarted","Data":"5465edfea8d9f6b714863d3da3b1f792e194062c3f9bd9c3edb519ef314b72e9"} Sep 15 21:26:58.489538 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.489532 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:26:58.489670 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.489553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" event={"ID":"fdb54a01-af83-4d22-9437-a97cb7169382","Type":"ContainerStarted","Data":"fabefb12314d4b11852966b9b8cbec74e7d702f33af1e504823836124c406f2f"} Sep 15 21:26:58.528783 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.527412 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" podStartSLOduration=1.5273938280000001 podStartE2EDuration="1.527393828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:58.526840755 +0000 UTC m=+67.866320207" watchObservedRunningTime="2026-09-15 21:26:58.527393828 +0000 UTC m=+67.866873284" Sep 15 21:26:58.563005 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.562620 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b5cd6b98-92qmw"] Sep 15 21:26:58.568164 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.568033 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59b5cd6b98-92qmw"] Sep 15 21:26:58.727936 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:58.727909 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sg6pl"] Sep 15 21:26:58.731313 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:26:58.731244 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72d9f28_2893_48a1_9089_8dcaf636f332.slice/crio-6353dcc8280599d7949bb4c5452a8bc64b51c6a0e26334600be07847bc06090b WatchSource:0}: Error finding container 6353dcc8280599d7949bb4c5452a8bc64b51c6a0e26334600be07847bc06090b: Status 404 returned error can't find the container with id 6353dcc8280599d7949bb4c5452a8bc64b51c6a0e26334600be07847bc06090b Sep 15 21:26:59.150059 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:59.149977 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065ab3b7-ad10-488f-9a97-5811583f9d82" path="/var/lib/kubelet/pods/065ab3b7-ad10-488f-9a97-5811583f9d82/volumes" Sep 15 21:26:59.501128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:59.501005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvr9k" event={"ID":"8ff6f6a8-1a38-427f-938c-36d5df919731","Type":"ContainerStarted","Data":"299a6c466f81414ff593a08e0a12b39066f378b11a7db3a23bcffba0c6892587"} Sep 15 21:26:59.501128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:59.501071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvr9k" event={"ID":"8ff6f6a8-1a38-427f-938c-36d5df919731","Type":"ContainerStarted","Data":"a6bbeeaa657397bf43f8bed8e3bdf84bfc23f2a0a89df1be86c9d5dca08ebb84"} Sep 15 21:26:59.503510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:59.503471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sg6pl" event={"ID":"a72d9f28-2893-48a1-9089-8dcaf636f332","Type":"ContainerStarted","Data":"6353dcc8280599d7949bb4c5452a8bc64b51c6a0e26334600be07847bc06090b"} Sep 15 21:26:59.519309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:26:59.519253 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lvr9k" podStartSLOduration=67.531739841 podStartE2EDuration="1m8.51923102s" totalImagesPullingTime="987.491179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:59.51834141 +0000 UTC m=+68.857820861" watchObservedRunningTime="2026-09-15 21:26:59.51923102 +0000 UTC m=+68.858710474" Sep 15 21:27:08.542165 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.542124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" event={"ID":"369e5023-7103-4182-866d-2a3dcd0ed422","Type":"ContainerStarted","Data":"54b5ef3b667b28b9ac64f396cf7d0ba2327f60b74a23be934c0ece0095133ee3"} Sep 15 21:27:08.542165 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.542171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" event={"ID":"369e5023-7103-4182-866d-2a3dcd0ed422","Type":"ContainerStarted","Data":"76da00f91f0f08e7c260790d4088520117f8868efd90797540de9aa8b39390b6"} Sep 15 21:27:08.542673 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.542197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" event={"ID":"369e5023-7103-4182-866d-2a3dcd0ed422","Type":"ContainerStarted","Data":"481491afbbe13ce6a2561233ee780e4b22a180a637dede441b76a764115ba2e7"} Sep 15 21:27:08.543774 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.543747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d7wjb" event={"ID":"b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f","Type":"ContainerStarted","Data":"574f2f1e951a568e343080de6982d5862e4e402483865ee135be4a3592568ff7"} Sep 15 21:27:08.545523 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.545499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sg6pl" event={"ID":"a72d9f28-2893-48a1-9089-8dcaf636f332","Type":"ContainerStarted","Data":"a3218ef80ee017fc3ed43ad98a32e5c54a2f4034070a9ca56a1c4c1409181f8e"} Sep 15 21:27:08.545599 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.545550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sg6pl" event={"ID":"a72d9f28-2893-48a1-9089-8dcaf636f332","Type":"ContainerStarted","Data":"d0a24c6e5639a71f74e3b758d7243915edab84de78d00bf00ecbe7c863086b15"} Sep 15 21:27:08.545638 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.545606 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sg6pl" Sep 15 21:27:08.546929 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.546911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" event={"ID":"89427796-61c1-4092-a49a-042f7dee3a8b","Type":"ContainerStarted","Data":"befc9d8fbd9c697b5f8cdde953619da0dc11c1702add370e11185ca9453446aa"} Sep 15 21:27:08.547127 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.547111 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:27:08.548417 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.548387 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gwdz" event={"ID":"4073f3d6-73af-480c-9184-5e6cf6542092","Type":"ContainerStarted","Data":"a689a2c1dc3d60aa1b5259171e454553195fd195f8844b7039f112750fd31292"} Sep 15 21:27:08.548511 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.548500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:27:08.549057 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.549043 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" Sep 15 21:27:08.562871 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.562832 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c4cc567b-bnprk" podStartSLOduration=1.747688446 podStartE2EDuration="11.562820111s" totalImagesPullingTime="9.815131665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:08.562312154 +0000 UTC m=+77.901791608" watchObservedRunningTime="2026-09-15 21:27:08.562820111 +0000 UTC m=+77.902299552" Sep 15 21:27:08.576399 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.576348 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d7wjb" podStartSLOduration=67.260225117 podStartE2EDuration="1m17.576333651s" totalImagesPullingTime="10.316108534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:08.576224109 +0000 UTC m=+77.915703562" watchObservedRunningTime="2026-09-15 21:27:08.576333651 +0000 UTC m=+77.915813103" Sep 15 21:27:08.593404 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.593363 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sg6pl" podStartSLOduration=34.522959575 podStartE2EDuration="43.593350108s" totalImagesPullingTime="9.070390533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:08.592873087 +0000 UTC m=+77.932352541" watchObservedRunningTime="2026-09-15 21:27:08.593350108 +0000 UTC m=+77.932829559" Sep 15 21:27:08.608007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.607877 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gwdz" podStartSLOduration=67.324640356 podStartE2EDuration="1m17.607867308s" totalImagesPullingTime="10.283226952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:08.607577496 +0000 UTC m=+77.947056950" watchObservedRunningTime="2026-09-15 21:27:08.607867308 +0000 UTC m=+77.947346759" Sep 15 21:27:08.623408 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:08.623363 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9fbcff75-mwsnp" podStartSLOduration=1.758731123 podStartE2EDuration="11.623344951s" totalImagesPullingTime="9.864613828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:08.622518075 +0000 UTC m=+77.961997527" watchObservedRunningTime="2026-09-15 21:27:08.623344951 +0000 UTC m=+77.962824404" Sep 15 21:27:10.703089 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.703054 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qdkxr"] Sep 15 21:27:10.706051 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.706029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.708699 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.708673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 21:27:10.708867 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.708852 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 21:27:10.709015 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.708991 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 21:27:10.709259 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.709242 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 21:27:10.709928 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.709902 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 21:27:10.710003 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.709933 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9lv77\"" Sep 15 21:27:10.710003 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.709905 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 21:27:10.774748 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.774715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-textfile\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.774748 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.774753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zm9\" (UniqueName: \"kubernetes.io/projected/32fd200d-709f-40db-8b1f-a0db9c35a906-kube-api-access-k2zm9\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.774781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.774912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-tls\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.774948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32fd200d-709f-40db-8b1f-a0db9c35a906-metrics-client-ca\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.774988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-sys\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775222 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.775041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775222 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.775091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-root\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.775222 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.775155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-wtmp\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.875793 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-root\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.875793 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-wtmp\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-textfile\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zm9\" (UniqueName: \"kubernetes.io/projected/32fd200d-709f-40db-8b1f-a0db9c35a906-kube-api-access-k2zm9\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-root\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-tls\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32fd200d-709f-40db-8b1f-a0db9c35a906-metrics-client-ca\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876024 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.875993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-sys\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.876037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.876034 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-wtmp\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.876133 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32fd200d-709f-40db-8b1f-a0db9c35a906-sys\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.876375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-textfile\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876531 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.876502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.876566 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.876541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32fd200d-709f-40db-8b1f-a0db9c35a906-metrics-client-ca\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.878436 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.878413 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.878680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.878593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32fd200d-709f-40db-8b1f-a0db9c35a906-node-exporter-tls\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:10.886041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:10.886023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zm9\" (UniqueName: \"kubernetes.io/projected/32fd200d-709f-40db-8b1f-a0db9c35a906-kube-api-access-k2zm9\") pod \"node-exporter-qdkxr\" (UID: \"32fd200d-709f-40db-8b1f-a0db9c35a906\") " pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:11.021779 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:11.021691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdkxr" Sep 15 21:27:11.032217 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:27:11.032180 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fd200d_709f_40db_8b1f_a0db9c35a906.slice/crio-d1d07eb910f62f3823f692576a19bf454e01e9d7ef528926f9f12db62ab77ace WatchSource:0}: Error finding container d1d07eb910f62f3823f692576a19bf454e01e9d7ef528926f9f12db62ab77ace: Status 404 returned error can't find the container with id d1d07eb910f62f3823f692576a19bf454e01e9d7ef528926f9f12db62ab77ace Sep 15 21:27:11.559954 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:11.559915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxr" event={"ID":"32fd200d-709f-40db-8b1f-a0db9c35a906","Type":"ContainerStarted","Data":"d1d07eb910f62f3823f692576a19bf454e01e9d7ef528926f9f12db62ab77ace"} Sep 15 21:27:12.567628 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:12.567538 2575 generic.go:356] "Generic (PLEG): container finished" podID="32fd200d-709f-40db-8b1f-a0db9c35a906" containerID="828905c53d3bbc762bee9b6547e4781ea5deb4b034aba95f153da8f517d5d103" exitCode=0 Sep 15 21:27:12.567628 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:12.567612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxr" event={"ID":"32fd200d-709f-40db-8b1f-a0db9c35a906","Type":"ContainerDied","Data":"828905c53d3bbc762bee9b6547e4781ea5deb4b034aba95f153da8f517d5d103"} Sep 15 21:27:13.573984 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:13.573944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxr" event={"ID":"32fd200d-709f-40db-8b1f-a0db9c35a906","Type":"ContainerStarted","Data":"e951803b882b47aa31c293867959ecbeef4f9b8235efb246d61187a9fa4d2eb5"} Sep 15 21:27:13.574483 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:13.574006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxr" event={"ID":"32fd200d-709f-40db-8b1f-a0db9c35a906","Type":"ContainerStarted","Data":"04ece1815fbd30efce2c1f397a0cf39fc223713eee4aa24d4d770e2a08e9cc4b"} Sep 15 21:27:13.592904 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:13.592859 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qdkxr" podStartSLOduration=2.267860414 podStartE2EDuration="3.592844826s" totalImagesPullingTime="1.256544593s" totalInitContainerRuntime="68.439819ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:13.591832198 +0000 UTC m=+82.931311649" watchObservedRunningTime="2026-09-15 21:27:13.592844826 +0000 UTC m=+82.932324316" Sep 15 21:27:18.555115 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:18.555066 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sg6pl" Sep 15 21:27:19.507645 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:19.507619 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-757c5b4b9d-7pz5c" Sep 15 21:27:39.554848 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:39.554819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gwdz" Sep 15 21:27:52.342326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:27:52.342296 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6rdt7_30c0de63-a264-4c60-8166-f0bdbfca7e0c/serve-healthcheck-canary/0.log" Sep 15 21:30:51.030928 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:30:51.030899 2575 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 21:31:26.760910 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.760866 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw"] Sep 15 21:31:26.762744 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.762730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:26.765364 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.765338 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cf7g2\"" Sep 15 21:31:26.765364 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.765351 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 21:31:26.765533 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.765363 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:31:26.771652 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.771629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw"] Sep 15 21:31:26.872192 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.872153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad56ee4b-84d5-4944-abb9-bcce76bd7c5f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5rbfw\" (UID: \"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:26.872360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.872215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngnc\" (UniqueName: \"kubernetes.io/projected/ad56ee4b-84d5-4944-abb9-bcce76bd7c5f-kube-api-access-gngnc\") pod \"openshift-lws-operator-fd8ccff4c-5rbfw\" (UID: \"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:26.972553 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.972520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad56ee4b-84d5-4944-abb9-bcce76bd7c5f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5rbfw\" (UID: \"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:26.972669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.972577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngnc\" (UniqueName: \"kubernetes.io/projected/ad56ee4b-84d5-4944-abb9-bcce76bd7c5f-kube-api-access-gngnc\") pod \"openshift-lws-operator-fd8ccff4c-5rbfw\" (UID: \"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:26.972882 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.972862 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad56ee4b-84d5-4944-abb9-bcce76bd7c5f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5rbfw\" (UID: \"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:26.983040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:26.983015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngnc\" (UniqueName: \"kubernetes.io/projected/ad56ee4b-84d5-4944-abb9-bcce76bd7c5f-kube-api-access-gngnc\") pod \"openshift-lws-operator-fd8ccff4c-5rbfw\" (UID: \"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:27.072246 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:27.072175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" Sep 15 21:31:27.343493 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:27.341988 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw"] Sep 15 21:31:27.347226 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:27.347208 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:31:27.419333 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:27.419302 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" event={"ID":"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f","Type":"ContainerStarted","Data":"a26c905c15ec2575ec6ad746188944e5bde77ad2421d91036157226893034b0f"} Sep 15 21:31:30.433317 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:30.433286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" event={"ID":"ad56ee4b-84d5-4944-abb9-bcce76bd7c5f","Type":"ContainerStarted","Data":"47a2130d77d994c408658642443224b7f9d5ab71b8dd57ef8d6afc9c38e6d3f6"} Sep 15 21:31:30.448903 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:30.448855 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5rbfw" podStartSLOduration=1.8326819319999998 podStartE2EDuration="4.448839471s" totalImagesPullingTime="2.616157539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:30.448066887 +0000 UTC m=+339.787546338" watchObservedRunningTime="2026-09-15 21:31:30.448839471 +0000 UTC m=+339.788318922" Sep 15 21:31:45.510245 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.510149 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc"] Sep 15 21:31:45.516541 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.516512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.519119 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.519066 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 21:31:45.519119 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.519081 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 21:31:45.519119 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.519091 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5f75r\"" Sep 15 21:31:45.519323 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.519217 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 21:31:45.519374 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.519350 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 21:31:45.529533 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.529511 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc"] Sep 15 21:31:45.604111 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.604078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7ab201-4302-420a-b401-6aa2d55b9447-tmp\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.604271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.604136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e7ab201-4302-420a-b401-6aa2d55b9447-manifests\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.604271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.604225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e7ab201-4302-420a-b401-6aa2d55b9447-webhook-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.604271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.604252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgksv\" (UniqueName: \"kubernetes.io/projected/5e7ab201-4302-420a-b401-6aa2d55b9447-kube-api-access-rgksv\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.604379 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.604310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e7ab201-4302-420a-b401-6aa2d55b9447-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.705507 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.705470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7ab201-4302-420a-b401-6aa2d55b9447-tmp\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.705507 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.705508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e7ab201-4302-420a-b401-6aa2d55b9447-manifests\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.705695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.705642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e7ab201-4302-420a-b401-6aa2d55b9447-webhook-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.705695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.705673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgksv\" (UniqueName: \"kubernetes.io/projected/5e7ab201-4302-420a-b401-6aa2d55b9447-kube-api-access-rgksv\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.705762 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.705712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e7ab201-4302-420a-b401-6aa2d55b9447-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.705921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.705903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e7ab201-4302-420a-b401-6aa2d55b9447-manifests\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.708110 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.708064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7ab201-4302-420a-b401-6aa2d55b9447-tmp\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.708230 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.708164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e7ab201-4302-420a-b401-6aa2d55b9447-webhook-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.708230 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.708213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e7ab201-4302-420a-b401-6aa2d55b9447-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.725109 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.725063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgksv\" (UniqueName: \"kubernetes.io/projected/5e7ab201-4302-420a-b401-6aa2d55b9447-kube-api-access-rgksv\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-884vc\" (UID: \"5e7ab201-4302-420a-b401-6aa2d55b9447\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:45.826355 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:45.826255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:46.101945 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:46.101871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc"] Sep 15 21:31:46.105506 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:31:46.105475 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7ab201_4302_420a_b401_6aa2d55b9447.slice/crio-ece17fb9e3c723efc51eeac31da89169a10225245577ea02e3a83a2809301d38 WatchSource:0}: Error finding container ece17fb9e3c723efc51eeac31da89169a10225245577ea02e3a83a2809301d38: Status 404 returned error can't find the container with id ece17fb9e3c723efc51eeac31da89169a10225245577ea02e3a83a2809301d38 Sep 15 21:31:46.487290 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:46.487250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" event={"ID":"5e7ab201-4302-420a-b401-6aa2d55b9447","Type":"ContainerStarted","Data":"ece17fb9e3c723efc51eeac31da89169a10225245577ea02e3a83a2809301d38"} Sep 15 21:31:49.502020 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:49.501936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" event={"ID":"5e7ab201-4302-420a-b401-6aa2d55b9447","Type":"ContainerStarted","Data":"bcc2092cd10b0209f45428b8d51abffdd9e33229cf12bdd9a9fe7e22f4a5b476"} Sep 15 21:31:50.507132 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:50.507080 2575 generic.go:356] "Generic (PLEG): container finished" podID="5e7ab201-4302-420a-b401-6aa2d55b9447" containerID="bcc2092cd10b0209f45428b8d51abffdd9e33229cf12bdd9a9fe7e22f4a5b476" exitCode=0 Sep 15 21:31:50.507501 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:50.507168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" event={"ID":"5e7ab201-4302-420a-b401-6aa2d55b9447","Type":"ContainerDied","Data":"bcc2092cd10b0209f45428b8d51abffdd9e33229cf12bdd9a9fe7e22f4a5b476"} Sep 15 21:31:51.512225 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:51.512192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" event={"ID":"5e7ab201-4302-420a-b401-6aa2d55b9447","Type":"ContainerStarted","Data":"e9fdddad31397f35be80afa5e7e238e2d978467b8c3aeb0e1c75964835f5f23a"} Sep 15 21:31:51.512606 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:51.512300 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:31:51.537482 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:31:51.537434 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" podStartSLOduration=1.925373628 podStartE2EDuration="6.53741822s" totalImagesPullingTime="3.529742708s" totalInitContainerRuntime="1.082301884s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:51.53535707 +0000 UTC m=+360.874836520" watchObservedRunningTime="2026-09-15 21:31:51.53741822 +0000 UTC m=+360.876897670" Sep 15 21:32:02.518391 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:02.518353 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-884vc" Sep 15 21:32:16.746746 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.746708 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh"] Sep 15 21:32:16.749891 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.749868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.753315 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.753294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 15 21:32:16.753588 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.753571 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 21:32:16.753643 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.753579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 15 21:32:16.754165 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.754152 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rfdsv\"" Sep 15 21:32:16.754461 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.754448 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 21:32:16.766713 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.766692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh"] Sep 15 21:32:16.830129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.830087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-tls-certs\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.830262 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.830154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-tmp\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.830262 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.830244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvcw\" (UniqueName: \"kubernetes.io/projected/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-kube-api-access-cpvcw\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.849757 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.849731 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rnkfz"] Sep 15 21:32:16.852725 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.852707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:16.854963 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.854941 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 15 21:32:16.855059 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.854967 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-flqrb\"" Sep 15 21:32:16.865246 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.865225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rnkfz"] Sep 15 21:32:16.931162 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.931130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-tls-certs\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.931330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.931196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-tmp\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.931330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.931224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec606442-9df8-4aa9-9cf5-6953444b7453-cert\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:16.931330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.931243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv2v2\" (UniqueName: \"kubernetes.io/projected/ec606442-9df8-4aa9-9cf5-6953444b7453-kube-api-access-fv2v2\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:16.931330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.931266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvcw\" (UniqueName: \"kubernetes.io/projected/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-kube-api-access-cpvcw\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.933525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.933495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-tmp\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.933717 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.933698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-tls-certs\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:16.939696 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:16.939672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvcw\" (UniqueName: \"kubernetes.io/projected/a52160bd-aebe-4ae4-bca5-8bc03a23b7de-kube-api-access-cpvcw\") pod \"kube-auth-proxy-d78dff5cc-l48jh\" (UID: \"a52160bd-aebe-4ae4-bca5-8bc03a23b7de\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:17.031912 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.031845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec606442-9df8-4aa9-9cf5-6953444b7453-cert\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:17.031912 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.031878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv2v2\" (UniqueName: \"kubernetes.io/projected/ec606442-9df8-4aa9-9cf5-6953444b7453-kube-api-access-fv2v2\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:17.032050 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:17.031985 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 15 21:32:17.032088 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:17.032057 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec606442-9df8-4aa9-9cf5-6953444b7453-cert podName:ec606442-9df8-4aa9-9cf5-6953444b7453 nodeName:}" failed. No retries permitted until 2026-09-15 21:32:17.53203993 +0000 UTC m=+386.871519358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ec606442-9df8-4aa9-9cf5-6953444b7453-cert") pod "odh-model-controller-7998c67dc4-rnkfz" (UID: "ec606442-9df8-4aa9-9cf5-6953444b7453") : secret "odh-model-controller-webhook-cert" not found Sep 15 21:32:17.040764 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.040742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv2v2\" (UniqueName: \"kubernetes.io/projected/ec606442-9df8-4aa9-9cf5-6953444b7453-kube-api-access-fv2v2\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:17.059539 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.059518 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" Sep 15 21:32:17.260890 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.260860 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jmh62"] Sep 15 21:32:17.264027 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.264009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.266428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.266405 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 15 21:32:17.275305 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.275280 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jmh62"] Sep 15 21:32:17.335791 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.335023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b703202-1f21-461a-8791-859ee3a573ba-tls-certs\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.335791 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.335156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzf4\" (UniqueName: \"kubernetes.io/projected/3b703202-1f21-461a-8791-859ee3a573ba-kube-api-access-rdzf4\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.336859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.336833 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh"] Sep 15 21:32:17.340409 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:32:17.340386 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52160bd_aebe_4ae4_bca5_8bc03a23b7de.slice/crio-b150af147c103bb851f30361ba5dddf175a5e79f7ad0c74479687dae52f69b2e WatchSource:0}: Error finding container b150af147c103bb851f30361ba5dddf175a5e79f7ad0c74479687dae52f69b2e: Status 404 returned error can't find the container with id b150af147c103bb851f30361ba5dddf175a5e79f7ad0c74479687dae52f69b2e Sep 15 21:32:17.436545 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.436515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzf4\" (UniqueName: \"kubernetes.io/projected/3b703202-1f21-461a-8791-859ee3a573ba-kube-api-access-rdzf4\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.436665 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.436552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b703202-1f21-461a-8791-859ee3a573ba-tls-certs\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.436706 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:17.436672 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 15 21:32:17.436739 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:17.436732 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b703202-1f21-461a-8791-859ee3a573ba-tls-certs podName:3b703202-1f21-461a-8791-859ee3a573ba nodeName:}" failed. No retries permitted until 2026-09-15 21:32:17.936719193 +0000 UTC m=+387.276198622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3b703202-1f21-461a-8791-859ee3a573ba-tls-certs") pod "model-serving-api-54d89bd5fc-jmh62" (UID: "3b703202-1f21-461a-8791-859ee3a573ba") : secret "model-serving-api-tls" not found Sep 15 21:32:17.446297 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.446275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzf4\" (UniqueName: \"kubernetes.io/projected/3b703202-1f21-461a-8791-859ee3a573ba-kube-api-access-rdzf4\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.537416 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.537382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec606442-9df8-4aa9-9cf5-6953444b7453-cert\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:17.539788 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.539761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec606442-9df8-4aa9-9cf5-6953444b7453-cert\") pod \"odh-model-controller-7998c67dc4-rnkfz\" (UID: \"ec606442-9df8-4aa9-9cf5-6953444b7453\") " pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:17.601873 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.601810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" event={"ID":"a52160bd-aebe-4ae4-bca5-8bc03a23b7de","Type":"ContainerStarted","Data":"b150af147c103bb851f30361ba5dddf175a5e79f7ad0c74479687dae52f69b2e"} Sep 15 21:32:17.761698 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.761671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:17.888585 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.888507 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l"] Sep 15 21:32:17.891820 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.891805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:17.894366 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.894344 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 15 21:32:17.894524 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.894376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 15 21:32:17.894524 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.894383 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 15 21:32:17.894687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.894640 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 21:32:17.894687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.894684 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9x555\"" Sep 15 21:32:17.906424 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.906401 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l"] Sep 15 21:32:17.940349 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.940324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b703202-1f21-461a-8791-859ee3a573ba-tls-certs\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:17.942959 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:17.942940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b703202-1f21-461a-8791-859ee3a573ba-tls-certs\") pod \"model-serving-api-54d89bd5fc-jmh62\" (UID: \"3b703202-1f21-461a-8791-859ee3a573ba\") " pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:18.031780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.031732 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rnkfz"] Sep 15 21:32:18.035252 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:32:18.035201 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec606442_9df8_4aa9_9cf5_6953444b7453.slice/crio-463f2f619b24bb16eb25c49f66f0ef1975ef4f00578eea88f199abfd9fc662a0 WatchSource:0}: Error finding container 463f2f619b24bb16eb25c49f66f0ef1975ef4f00578eea88f199abfd9fc662a0: Status 404 returned error can't find the container with id 463f2f619b24bb16eb25c49f66f0ef1975ef4f00578eea88f199abfd9fc662a0 Sep 15 21:32:18.041025 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.040948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.041093 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.041059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.041177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.041148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.041229 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.041179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.041229 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.041218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.041341 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.041249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.041341 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.041281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nf8m\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-kube-api-access-5nf8m\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142051 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.141981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142051 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nf8m\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-kube-api-access-5nf8m\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.142807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.142755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.144926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.144883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.145020 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.144998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.145076 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.145020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.145591 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.145569 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.150532 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.150505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nf8m\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-kube-api-access-5nf8m\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.150532 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.150526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x2h9l\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.174393 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.174366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:18.200376 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.200342 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:18.463415 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.463333 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jmh62"] Sep 15 21:32:18.467172 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:32:18.467146 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b703202_1f21_461a_8791_859ee3a573ba.slice/crio-a3d8460cbe3c83fcec48587f71d0f1c54edefa1bb460aadbfae72b2f94d4c8e4 WatchSource:0}: Error finding container a3d8460cbe3c83fcec48587f71d0f1c54edefa1bb460aadbfae72b2f94d4c8e4: Status 404 returned error can't find the container with id a3d8460cbe3c83fcec48587f71d0f1c54edefa1bb460aadbfae72b2f94d4c8e4 Sep 15 21:32:18.495289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.495264 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l"] Sep 15 21:32:18.498916 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:32:18.498882 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2b635f_644e_488d_9dc4_5e33cfa7dd38.slice/crio-2d9e8a6614c1a3d52e058fcaac144f9f67c0e3b3b55a58f92466b06a121fd85c WatchSource:0}: Error finding container 2d9e8a6614c1a3d52e058fcaac144f9f67c0e3b3b55a58f92466b06a121fd85c: Status 404 returned error can't find the container with id 2d9e8a6614c1a3d52e058fcaac144f9f67c0e3b3b55a58f92466b06a121fd85c Sep 15 21:32:18.608265 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.608226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" event={"ID":"ec606442-9df8-4aa9-9cf5-6953444b7453","Type":"ContainerStarted","Data":"463f2f619b24bb16eb25c49f66f0ef1975ef4f00578eea88f199abfd9fc662a0"} Sep 15 21:32:18.609641 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.609612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" event={"ID":"0c2b635f-644e-488d-9dc4-5e33cfa7dd38","Type":"ContainerStarted","Data":"2d9e8a6614c1a3d52e058fcaac144f9f67c0e3b3b55a58f92466b06a121fd85c"} Sep 15 21:32:18.610971 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:18.610946 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" event={"ID":"3b703202-1f21-461a-8791-859ee3a573ba","Type":"ContainerStarted","Data":"a3d8460cbe3c83fcec48587f71d0f1c54edefa1bb460aadbfae72b2f94d4c8e4"} Sep 15 21:32:22.170257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.170079 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 15 21:32:22.170257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.170177 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 15 21:32:22.631957 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.631905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" event={"ID":"3b703202-1f21-461a-8791-859ee3a573ba","Type":"ContainerStarted","Data":"67acf2b0b1f94873b05c773a34b562be6528f0e267678b539fabf9e94ec094ea"} Sep 15 21:32:22.632169 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.632005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:22.634042 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.634012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" event={"ID":"a52160bd-aebe-4ae4-bca5-8bc03a23b7de","Type":"ContainerStarted","Data":"f7d064053c08643274cf1b1648ef7f0ff6d5194e3fd009a1fa3d836f5175644f"} Sep 15 21:32:22.636052 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.636021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" event={"ID":"0c2b635f-644e-488d-9dc4-5e33cfa7dd38","Type":"ContainerStarted","Data":"1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5"} Sep 15 21:32:22.636236 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.636219 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:22.649425 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.649378 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" podStartSLOduration=1.9492485240000001 podStartE2EDuration="5.64935648s" totalImagesPullingTime="3.700107956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:22.648543311 +0000 UTC m=+391.988022764" watchObservedRunningTime="2026-09-15 21:32:22.64935648 +0000 UTC m=+391.988835933" Sep 15 21:32:22.667150 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:22.667087 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-l48jh" podStartSLOduration=1.844502241 podStartE2EDuration="6.667071735s" totalImagesPullingTime="4.822569494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:22.665591582 +0000 UTC m=+392.005071036" watchObservedRunningTime="2026-09-15 21:32:22.667071735 +0000 UTC m=+392.006551187" Sep 15 21:32:23.646313 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:23.646276 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x2h9l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:32:23.646771 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:23.646335 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" podUID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:32:25.653211 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:25.653178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" event={"ID":"ec606442-9df8-4aa9-9cf5-6953444b7453","Type":"ContainerStarted","Data":"da96be0dcf0e9350e647c941651500dced04f7f3c9c1b69268689961753f4a0e"} Sep 15 21:32:25.653595 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:25.653249 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:32:25.699875 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:25.699823 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" podStartSLOduration=3.1182528290000002 podStartE2EDuration="9.699806104s" totalImagesPullingTime="6.581553275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:25.696366558 +0000 UTC m=+395.035846007" watchObservedRunningTime="2026-09-15 21:32:25.699806104 +0000 UTC m=+395.039285554" Sep 15 21:32:25.701128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:25.701057 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" podStartSLOduration=5.032726703 podStartE2EDuration="8.701047095s" totalImagesPullingTime="3.668320392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:22.720921732 +0000 UTC m=+392.060401184" watchObservedRunningTime="2026-09-15 21:32:25.701047095 +0000 UTC m=+395.040526546" Sep 15 21:32:26.455521 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.455489 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s"] Sep 15 21:32:26.457649 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.457626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.460938 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.460915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bbnjh\"" Sep 15 21:32:26.461941 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.461917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 15 21:32:26.467551 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.467530 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv"] Sep 15 21:32:26.469513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.469498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:26.472719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.472703 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 15 21:32:26.473262 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.473242 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rz44h\"" Sep 15 21:32:26.477035 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.477012 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s"] Sep 15 21:32:26.482594 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.482571 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv"] Sep 15 21:32:26.613383 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.613349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3642b6f-2bbb-4075-b40c-1ebba6842f57-cert\") pod \"kserve-controller-manager-d6f4b65b8-fdd6s\" (UID: \"c3642b6f-2bbb-4075-b40c-1ebba6842f57\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.613538 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.613394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4spg\" (UniqueName: \"kubernetes.io/projected/c582fa99-30ba-455f-b3e4-92ff4fb770e0-kube-api-access-m4spg\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:26.613538 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.613467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcj5w\" (UniqueName: \"kubernetes.io/projected/c3642b6f-2bbb-4075-b40c-1ebba6842f57-kube-api-access-vcj5w\") pod \"kserve-controller-manager-d6f4b65b8-fdd6s\" (UID: \"c3642b6f-2bbb-4075-b40c-1ebba6842f57\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.613611 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.613552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c582fa99-30ba-455f-b3e4-92ff4fb770e0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:26.644867 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.644841 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:26.713958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.713875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcj5w\" (UniqueName: \"kubernetes.io/projected/c3642b6f-2bbb-4075-b40c-1ebba6842f57-kube-api-access-vcj5w\") pod \"kserve-controller-manager-d6f4b65b8-fdd6s\" (UID: \"c3642b6f-2bbb-4075-b40c-1ebba6842f57\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.714453 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.713978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c582fa99-30ba-455f-b3e4-92ff4fb770e0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:26.714453 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:26.714272 2575 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 15 21:32:26.714453 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:26.714333 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582fa99-30ba-455f-b3e4-92ff4fb770e0-cert podName:c582fa99-30ba-455f-b3e4-92ff4fb770e0 nodeName:}" failed. No retries permitted until 2026-09-15 21:32:27.214311983 +0000 UTC m=+396.553791415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c582fa99-30ba-455f-b3e4-92ff4fb770e0-cert") pod "llmisvc-controller-manager-777c6bd7d5-rdqbv" (UID: "c582fa99-30ba-455f-b3e4-92ff4fb770e0") : secret "llmisvc-webhook-server-cert" not found Sep 15 21:32:26.714453 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.714405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3642b6f-2bbb-4075-b40c-1ebba6842f57-cert\") pod \"kserve-controller-manager-d6f4b65b8-fdd6s\" (UID: \"c3642b6f-2bbb-4075-b40c-1ebba6842f57\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.714453 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.714442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4spg\" (UniqueName: \"kubernetes.io/projected/c582fa99-30ba-455f-b3e4-92ff4fb770e0-kube-api-access-m4spg\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:26.717053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.717023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3642b6f-2bbb-4075-b40c-1ebba6842f57-cert\") pod \"kserve-controller-manager-d6f4b65b8-fdd6s\" (UID: \"c3642b6f-2bbb-4075-b40c-1ebba6842f57\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.733412 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.733376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4spg\" (UniqueName: \"kubernetes.io/projected/c582fa99-30ba-455f-b3e4-92ff4fb770e0-kube-api-access-m4spg\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:26.733630 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.733603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcj5w\" (UniqueName: \"kubernetes.io/projected/c3642b6f-2bbb-4075-b40c-1ebba6842f57-kube-api-access-vcj5w\") pod \"kserve-controller-manager-d6f4b65b8-fdd6s\" (UID: \"c3642b6f-2bbb-4075-b40c-1ebba6842f57\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:26.770483 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:26.770454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:27.041881 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:27.041757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s"] Sep 15 21:32:27.044503 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:32:27.044476 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3642b6f_2bbb_4075_b40c_1ebba6842f57.slice/crio-bca839a2bd6770120c6861b193cd336b39097aa1ce8554e31f836d61875d5502 WatchSource:0}: Error finding container bca839a2bd6770120c6861b193cd336b39097aa1ce8554e31f836d61875d5502: Status 404 returned error can't find the container with id bca839a2bd6770120c6861b193cd336b39097aa1ce8554e31f836d61875d5502 Sep 15 21:32:27.218864 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:27.218832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c582fa99-30ba-455f-b3e4-92ff4fb770e0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:27.221260 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:27.221233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c582fa99-30ba-455f-b3e4-92ff4fb770e0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdqbv\" (UID: \"c582fa99-30ba-455f-b3e4-92ff4fb770e0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:27.381746 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:27.381664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:27.662697 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:27.662660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" event={"ID":"c3642b6f-2bbb-4075-b40c-1ebba6842f57","Type":"ContainerStarted","Data":"bca839a2bd6770120c6861b193cd336b39097aa1ce8554e31f836d61875d5502"} Sep 15 21:32:27.672573 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:27.672490 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv"] Sep 15 21:32:27.674501 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:32:27.674477 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc582fa99_30ba_455f_b3e4_92ff4fb770e0.slice/crio-378da985b7b4dc0cf7b47fa706a60bb55357b87f151214f60e3d89503af2a70a WatchSource:0}: Error finding container 378da985b7b4dc0cf7b47fa706a60bb55357b87f151214f60e3d89503af2a70a: Status 404 returned error can't find the container with id 378da985b7b4dc0cf7b47fa706a60bb55357b87f151214f60e3d89503af2a70a Sep 15 21:32:28.667989 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:28.667954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" event={"ID":"c582fa99-30ba-455f-b3e4-92ff4fb770e0","Type":"ContainerStarted","Data":"378da985b7b4dc0cf7b47fa706a60bb55357b87f151214f60e3d89503af2a70a"} Sep 15 21:32:28.881296 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:28.881260 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l"] Sep 15 21:32:28.881591 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:28.881553 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" podUID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" containerName="discovery" containerID="cri-o://1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5" gracePeriod=30 Sep 15 21:32:29.130322 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.130300 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:29.235477 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235438 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-local-certs\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.235618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235500 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-ca-configmap\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.235618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235549 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-dns-cert\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.235618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-kubeconfig\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.235850 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-cacerts\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.235850 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235658 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nf8m\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-kube-api-access-5nf8m\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.235850 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235687 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-token\") pod \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\" (UID: \"0c2b635f-644e-488d-9dc4-5e33cfa7dd38\") " Sep 15 21:32:29.236016 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.235897 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:32:29.238571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.238527 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:32:29.238696 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.238596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:32:29.238970 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.238930 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-local-certs" (OuterVolumeSpecName: "local-certs") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:32:29.239048 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.238982 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:32:29.239233 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.239210 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-kube-api-access-5nf8m" (OuterVolumeSpecName: "kube-api-access-5nf8m") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "kube-api-access-5nf8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:32:29.239523 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.239497 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-cacerts" (OuterVolumeSpecName: "cacerts") pod "0c2b635f-644e-488d-9dc4-5e33cfa7dd38" (UID: "0c2b635f-644e-488d-9dc4-5e33cfa7dd38"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:32:29.336716 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336635 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-local-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.336716 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336670 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-ca-configmap\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.336716 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336684 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-csr-dns-cert\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.336716 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336697 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-kubeconfig\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.336716 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336710 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-cacerts\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.337041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336722 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nf8m\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-kube-api-access-5nf8m\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.337041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.336736 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2b635f-644e-488d-9dc4-5e33cfa7dd38-istio-token\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:32:29.674011 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.673973 2575 generic.go:356] "Generic (PLEG): container finished" podID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" containerID="1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5" exitCode=0 Sep 15 21:32:29.674500 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.674027 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" Sep 15 21:32:29.674500 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.674059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" event={"ID":"0c2b635f-644e-488d-9dc4-5e33cfa7dd38","Type":"ContainerDied","Data":"1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5"} Sep 15 21:32:29.674500 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.674136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l" event={"ID":"0c2b635f-644e-488d-9dc4-5e33cfa7dd38","Type":"ContainerDied","Data":"2d9e8a6614c1a3d52e058fcaac144f9f67c0e3b3b55a58f92466b06a121fd85c"} Sep 15 21:32:29.674500 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.674172 2575 scope.go:117] "RemoveContainer" containerID="1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5" Sep 15 21:32:29.685774 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.685758 2575 scope.go:117] "RemoveContainer" containerID="1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5" Sep 15 21:32:29.686152 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:32:29.686119 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5\": container with ID starting with 1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5 not found: ID does not exist" containerID="1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5" Sep 15 21:32:29.686261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.686169 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5"} err="failed to get container status \"1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5\": rpc error: code = NotFound desc = could not find container \"1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5\": container with ID starting with 1e69d1d957327e30cef0d6fbe8ba2fbba0eb1004d7f748a11193f5cce43e95d5 not found: ID does not exist" Sep 15 21:32:29.725149 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.725089 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l"] Sep 15 21:32:29.730938 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:29.730883 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x2h9l"] Sep 15 21:32:31.146065 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.146036 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" path="/var/lib/kubelet/pods/0c2b635f-644e-488d-9dc4-5e33cfa7dd38/volumes" Sep 15 21:32:31.685902 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.685860 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" event={"ID":"c3642b6f-2bbb-4075-b40c-1ebba6842f57","Type":"ContainerStarted","Data":"06f67c165d276164c88a156e50b2f15066e79ab34c9678f70735a991cb5d4bcc"} Sep 15 21:32:31.686062 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.685986 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:32:31.687495 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.687467 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" event={"ID":"c582fa99-30ba-455f-b3e4-92ff4fb770e0","Type":"ContainerStarted","Data":"388b979165809eec337b40e63d5bea29e2a5fbd9837e51a2023d577d8ba1f93e"} Sep 15 21:32:31.687597 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.687511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:32:31.702387 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.702351 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" podStartSLOduration=1.54460468 podStartE2EDuration="5.702339425s" totalImagesPullingTime="4.157734745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:31.70164765 +0000 UTC m=+401.041127093" watchObservedRunningTime="2026-09-15 21:32:31.702339425 +0000 UTC m=+401.041818902" Sep 15 21:32:31.717335 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:31.717294 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" podStartSLOduration=2.191565784 podStartE2EDuration="5.717281512s" totalImagesPullingTime="3.525715728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:31.716559179 +0000 UTC m=+401.056038633" watchObservedRunningTime="2026-09-15 21:32:31.717281512 +0000 UTC m=+401.056760964" Sep 15 21:32:33.648378 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:33.648347 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jmh62" Sep 15 21:32:36.659453 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:32:36.659423 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rnkfz" Sep 15 21:33:02.694571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:02.694542 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdqbv" Sep 15 21:33:02.697519 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:02.697497 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fdd6s" Sep 15 21:33:34.109974 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.109936 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79"] Sep 15 21:33:34.110441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.110260 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" containerName="discovery" Sep 15 21:33:34.110441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.110273 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" containerName="discovery" Sep 15 21:33:34.110441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.110324 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2b635f-644e-488d-9dc4-5e33cfa7dd38" containerName="discovery" Sep 15 21:33:34.112441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.112425 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:34.115036 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.115007 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 21:33:34.115172 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.115015 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s5bgp\"" Sep 15 21:33:34.115989 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.115975 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 21:33:34.124325 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.124307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79"] Sep 15 21:33:34.204480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.204444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2lch\" (UniqueName: \"kubernetes.io/projected/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1-kube-api-access-l2lch\") pod \"limitador-operator-controller-manager-74495c69f8-zpg79\" (UID: \"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:34.304880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.304848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2lch\" (UniqueName: \"kubernetes.io/projected/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1-kube-api-access-l2lch\") pod \"limitador-operator-controller-manager-74495c69f8-zpg79\" (UID: \"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:34.317175 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.317153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2lch\" (UniqueName: \"kubernetes.io/projected/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1-kube-api-access-l2lch\") pod \"limitador-operator-controller-manager-74495c69f8-zpg79\" (UID: \"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:34.421980 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.421953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:34.693852 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.693658 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79"] Sep 15 21:33:34.696602 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:33:34.696575 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f73eb91_27d0_4873_aa06_e7a5cf98c8b1.slice/crio-a04c266972164689d9e6f6646648b9e6f2d54828b669176549dea5ed2e37c97d WatchSource:0}: Error finding container a04c266972164689d9e6f6646648b9e6f2d54828b669176549dea5ed2e37c97d: Status 404 returned error can't find the container with id a04c266972164689d9e6f6646648b9e6f2d54828b669176549dea5ed2e37c97d Sep 15 21:33:34.944651 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:34.944572 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" event={"ID":"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1","Type":"ContainerStarted","Data":"a04c266972164689d9e6f6646648b9e6f2d54828b669176549dea5ed2e37c97d"} Sep 15 21:33:37.239416 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.239379 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb"] Sep 15 21:33:37.242854 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.242830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:37.245594 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.245563 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 15 21:33:37.245700 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.245575 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nzcns\"" Sep 15 21:33:37.255787 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.255733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb"] Sep 15 21:33:37.329573 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.329545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5nh\" (UniqueName: \"kubernetes.io/projected/d6ae0828-b03e-4601-86dc-e2f77ec60824-kube-api-access-cv5nh\") pod \"dns-operator-controller-manager-7b84cd7c9c-29tdb\" (UID: \"d6ae0828-b03e-4601-86dc-e2f77ec60824\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:37.430837 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.430812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5nh\" (UniqueName: \"kubernetes.io/projected/d6ae0828-b03e-4601-86dc-e2f77ec60824-kube-api-access-cv5nh\") pod \"dns-operator-controller-manager-7b84cd7c9c-29tdb\" (UID: \"d6ae0828-b03e-4601-86dc-e2f77ec60824\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:37.438849 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.438832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5nh\" (UniqueName: \"kubernetes.io/projected/d6ae0828-b03e-4601-86dc-e2f77ec60824-kube-api-access-cv5nh\") pod \"dns-operator-controller-manager-7b84cd7c9c-29tdb\" (UID: \"d6ae0828-b03e-4601-86dc-e2f77ec60824\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:37.556213 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.556129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:37.836166 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.836064 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb"] Sep 15 21:33:37.840419 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:33:37.840393 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ae0828_b03e_4601_86dc_e2f77ec60824.slice/crio-4758613b4f7e1fa4b751011560340bf05de9c90d1c826fab3826bbd36fccdd69 WatchSource:0}: Error finding container 4758613b4f7e1fa4b751011560340bf05de9c90d1c826fab3826bbd36fccdd69: Status 404 returned error can't find the container with id 4758613b4f7e1fa4b751011560340bf05de9c90d1c826fab3826bbd36fccdd69 Sep 15 21:33:37.959913 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.959882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" event={"ID":"d6ae0828-b03e-4601-86dc-e2f77ec60824","Type":"ContainerStarted","Data":"4758613b4f7e1fa4b751011560340bf05de9c90d1c826fab3826bbd36fccdd69"} Sep 15 21:33:37.961563 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.961537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" event={"ID":"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1","Type":"ContainerStarted","Data":"0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466"} Sep 15 21:33:37.961684 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.961657 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:37.981637 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:37.981592 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" podStartSLOduration=1.3193012020000001 podStartE2EDuration="3.981578439s" totalImagesPullingTime="2.662277237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:37.980193304 +0000 UTC m=+467.319672756" watchObservedRunningTime="2026-09-15 21:33:37.981578439 +0000 UTC m=+467.321057889" Sep 15 21:33:39.972615 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:39.972529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" event={"ID":"d6ae0828-b03e-4601-86dc-e2f77ec60824","Type":"ContainerStarted","Data":"d89231164ec3138f58f629aaec84f9131953d3ed72dcb39423697942b163537e"} Sep 15 21:33:39.972976 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:39.972667 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:39.993006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:39.992957 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" podStartSLOduration=1.154937041 podStartE2EDuration="2.992941544s" totalImagesPullingTime="1.838004503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:39.991443086 +0000 UTC m=+469.330922537" watchObservedRunningTime="2026-09-15 21:33:39.992941544 +0000 UTC m=+469.332420994" Sep 15 21:33:42.173731 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.173692 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc"] Sep 15 21:33:42.177078 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.177057 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.179295 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.179276 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-52mqn\"" Sep 15 21:33:42.187146 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.187123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc"] Sep 15 21:33:42.268152 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.268086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d49b95-af26-4035-9c96-7a362deb5bf5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.268152 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.268158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72t2\" (UniqueName: \"kubernetes.io/projected/e7d49b95-af26-4035-9c96-7a362deb5bf5-kube-api-access-b72t2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.368615 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.368573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d49b95-af26-4035-9c96-7a362deb5bf5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.368615 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.368618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72t2\" (UniqueName: \"kubernetes.io/projected/e7d49b95-af26-4035-9c96-7a362deb5bf5-kube-api-access-b72t2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.368958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.368938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d49b95-af26-4035-9c96-7a362deb5bf5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.376494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.376468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72t2\" (UniqueName: \"kubernetes.io/projected/e7d49b95-af26-4035-9c96-7a362deb5bf5-kube-api-access-b72t2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.487175 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.487067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:42.764667 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.764541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc"] Sep 15 21:33:42.767672 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:33:42.767635 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d49b95_af26_4035_9c96_7a362deb5bf5.slice/crio-6dc93d6f8f6fabf806848be706bb3bc83aa1d0960c900da0b0b84a8fa43fe8b6 WatchSource:0}: Error finding container 6dc93d6f8f6fabf806848be706bb3bc83aa1d0960c900da0b0b84a8fa43fe8b6: Status 404 returned error can't find the container with id 6dc93d6f8f6fabf806848be706bb3bc83aa1d0960c900da0b0b84a8fa43fe8b6 Sep 15 21:33:42.985702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:42.985673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" event={"ID":"e7d49b95-af26-4035-9c96-7a362deb5bf5","Type":"ContainerStarted","Data":"6dc93d6f8f6fabf806848be706bb3bc83aa1d0960c900da0b0b84a8fa43fe8b6"} Sep 15 21:33:45.208650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.208618 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5lr4s"] Sep 15 21:33:45.211203 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.211180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:33:45.213596 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.213574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fqj2w\"" Sep 15 21:33:45.222144 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.222089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5lr4s"] Sep 15 21:33:45.294274 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.294235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmv8\" (UniqueName: \"kubernetes.io/projected/fb9954ae-c239-46bc-b12d-9f883a1a6af3-kube-api-access-xjmv8\") pod \"authorino-operator-76b478bd78-5lr4s\" (UID: \"fb9954ae-c239-46bc-b12d-9f883a1a6af3\") " pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:33:45.395333 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.395292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmv8\" (UniqueName: \"kubernetes.io/projected/fb9954ae-c239-46bc-b12d-9f883a1a6af3-kube-api-access-xjmv8\") pod \"authorino-operator-76b478bd78-5lr4s\" (UID: \"fb9954ae-c239-46bc-b12d-9f883a1a6af3\") " pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:33:45.404687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.404653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmv8\" (UniqueName: \"kubernetes.io/projected/fb9954ae-c239-46bc-b12d-9f883a1a6af3-kube-api-access-xjmv8\") pod \"authorino-operator-76b478bd78-5lr4s\" (UID: \"fb9954ae-c239-46bc-b12d-9f883a1a6af3\") " pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:33:45.528501 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:45.528411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:33:46.991738 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:46.991654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5lr4s"] Sep 15 21:33:46.994034 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:33:46.994007 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9954ae_c239_46bc_b12d_9f883a1a6af3.slice/crio-22f7aef0c612b11c36d9cf50f5f2935faa372d018203355ea2f1f4de40cc12a8 WatchSource:0}: Error finding container 22f7aef0c612b11c36d9cf50f5f2935faa372d018203355ea2f1f4de40cc12a8: Status 404 returned error can't find the container with id 22f7aef0c612b11c36d9cf50f5f2935faa372d018203355ea2f1f4de40cc12a8 Sep 15 21:33:47.005571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:47.005537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" event={"ID":"fb9954ae-c239-46bc-b12d-9f883a1a6af3","Type":"ContainerStarted","Data":"22f7aef0c612b11c36d9cf50f5f2935faa372d018203355ea2f1f4de40cc12a8"} Sep 15 21:33:47.007225 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:47.007190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" event={"ID":"e7d49b95-af26-4035-9c96-7a362deb5bf5","Type":"ContainerStarted","Data":"125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a"} Sep 15 21:33:47.007339 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:47.007323 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:47.027958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:47.027913 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" podStartSLOduration=1.032189023 podStartE2EDuration="5.027898487s" totalImagesPullingTime="3.995709464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:47.026175895 +0000 UTC m=+476.365655346" watchObservedRunningTime="2026-09-15 21:33:47.027898487 +0000 UTC m=+476.367377937" Sep 15 21:33:48.969113 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:48.969008 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:33:49.017414 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:49.017376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" event={"ID":"fb9954ae-c239-46bc-b12d-9f883a1a6af3","Type":"ContainerStarted","Data":"37994671b4cc6760a76728ed9fef6a773155283c0f20d75bf9fdef79d5981146"} Sep 15 21:33:49.017573 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:49.017512 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:33:50.978775 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:50.978746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-29tdb" Sep 15 21:33:51.002327 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:51.002275 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" podStartSLOduration=4.29825616 podStartE2EDuration="6.002257162s" totalImagesPullingTime="1.704001002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:49.045951427 +0000 UTC m=+478.385430879" watchObservedRunningTime="2026-09-15 21:33:51.002257162 +0000 UTC m=+480.341736612" Sep 15 21:33:58.013950 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:58.013918 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:33:59.668881 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.668849 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp"] Sep 15 21:33:59.671295 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.671280 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:33:59.685145 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.685121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp"] Sep 15 21:33:59.812988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.812956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7057d956-e6b7-4acb-bbc0-bd9c7d874524-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:33:59.812988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.812999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjzs\" (UniqueName: \"kubernetes.io/projected/7057d956-e6b7-4acb-bbc0-bd9c7d874524-kube-api-access-njjzs\") pod \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:33:59.818665 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.818638 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp"] Sep 15 21:33:59.818894 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:33:59.818872 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-njjzs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" podUID="7057d956-e6b7-4acb-bbc0-bd9c7d874524" Sep 15 21:33:59.914425 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.914371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7057d956-e6b7-4acb-bbc0-bd9c7d874524-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:33:59.914588 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.914437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjzs\" (UniqueName: \"kubernetes.io/projected/7057d956-e6b7-4acb-bbc0-bd9c7d874524-kube-api-access-njjzs\") pod \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:33:59.914761 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.914741 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7057d956-e6b7-4acb-bbc0-bd9c7d874524-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:33:59.922467 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:33:59.922416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjzs\" (UniqueName: \"kubernetes.io/projected/7057d956-e6b7-4acb-bbc0-bd9c7d874524-kube-api-access-njjzs\") pod \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:34:00.024244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.024214 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-5lr4s" Sep 15 21:34:00.068320 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.068294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:34:00.074728 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.074704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:34:00.186422 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.186329 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp"] Sep 15 21:34:00.198163 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.198130 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp"] Sep 15 21:34:00.206374 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.206351 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc"] Sep 15 21:34:00.206537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.206513 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" containerName="manager" containerID="cri-o://125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a" gracePeriod=2 Sep 15 21:34:00.216264 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.216239 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc"] Sep 15 21:34:00.216899 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.216444 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njjzs\" (UniqueName: \"kubernetes.io/projected/7057d956-e6b7-4acb-bbc0-bd9c7d874524-kube-api-access-njjzs\") pod \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " Sep 15 21:34:00.216899 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.216490 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7057d956-e6b7-4acb-bbc0-bd9c7d874524-extensions-socket-volume\") pod \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\" (UID: \"7057d956-e6b7-4acb-bbc0-bd9c7d874524\") " Sep 15 21:34:00.216899 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.216799 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7057d956-e6b7-4acb-bbc0-bd9c7d874524-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7057d956-e6b7-4acb-bbc0-bd9c7d874524" (UID: "7057d956-e6b7-4acb-bbc0-bd9c7d874524"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:00.220793 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.220758 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7057d956-e6b7-4acb-bbc0-bd9c7d874524-kube-api-access-njjzs" (OuterVolumeSpecName: "kube-api-access-njjzs") pod "7057d956-e6b7-4acb-bbc0-bd9c7d874524" (UID: "7057d956-e6b7-4acb-bbc0-bd9c7d874524"). InnerVolumeSpecName "kube-api-access-njjzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:00.226359 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.226322 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e7d49b95-af26-4035-9c96-7a362deb5bf5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\": pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.232525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.232504 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl"] Sep 15 21:34:00.232807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.232795 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" containerName="manager" Sep 15 21:34:00.232853 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.232808 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" containerName="manager" Sep 15 21:34:00.232891 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.232861 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" containerName="manager" Sep 15 21:34:00.235221 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.235202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.237227 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.237199 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.248797 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.248775 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl"] Sep 15 21:34:00.257042 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.257021 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79"] Sep 15 21:34:00.257724 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.257691 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" containerName="manager" containerID="cri-o://0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466" gracePeriod=2 Sep 15 21:34:00.268730 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.268705 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79"] Sep 15 21:34:00.278244 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.278089 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.280023 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.279989 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:00.285061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.285037 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c"] Sep 15 21:34:00.285691 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.285670 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" containerName="manager" Sep 15 21:34:00.285691 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.285694 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" containerName="manager" Sep 15 21:34:00.285847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.285774 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" containerName="manager" Sep 15 21:34:00.291327 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.291311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:00.293688 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.293656 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.305574 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.305545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c"] Sep 15 21:34:00.317289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.317255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sttkl\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.317439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.317304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qnl\" (UniqueName: \"kubernetes.io/projected/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-kube-api-access-g4qnl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sttkl\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.317439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.317409 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njjzs\" (UniqueName: \"kubernetes.io/projected/7057d956-e6b7-4acb-bbc0-bd9c7d874524-kube-api-access-njjzs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:00.317439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.317428 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7057d956-e6b7-4acb-bbc0-bd9c7d874524-extensions-socket-volume\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:00.319620 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.319589 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:00.418472 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.418427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sttkl\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.418652 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.418483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qnl\" (UniqueName: \"kubernetes.io/projected/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-kube-api-access-g4qnl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sttkl\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.418652 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.418537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw6n\" (UniqueName: \"kubernetes.io/projected/32498c46-5964-4d1d-8f30-5aac8b7245f6-kube-api-access-lqw6n\") pod \"limitador-operator-controller-manager-74495c69f8-kch8c\" (UID: \"32498c46-5964-4d1d-8f30-5aac8b7245f6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:00.418905 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.418879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sttkl\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.429699 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.429664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qnl\" (UniqueName: \"kubernetes.io/projected/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-kube-api-access-g4qnl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sttkl\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.500434 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.500410 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.502441 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.502414 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.505819 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.505794 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:00.507006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.506991 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:00.508932 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.508912 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:00.511122 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:00.511086 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:00.519479 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.519458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw6n\" (UniqueName: \"kubernetes.io/projected/32498c46-5964-4d1d-8f30-5aac8b7245f6-kube-api-access-lqw6n\") pod \"limitador-operator-controller-manager-74495c69f8-kch8c\" (UID: \"32498c46-5964-4d1d-8f30-5aac8b7245f6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:00.535646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.535623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw6n\" (UniqueName: \"kubernetes.io/projected/32498c46-5964-4d1d-8f30-5aac8b7245f6-kube-api-access-lqw6n\") pod \"limitador-operator-controller-manager-74495c69f8-kch8c\" (UID: \"32498c46-5964-4d1d-8f30-5aac8b7245f6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:00.620414 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.620379 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b72t2\" (UniqueName: \"kubernetes.io/projected/e7d49b95-af26-4035-9c96-7a362deb5bf5-kube-api-access-b72t2\") pod \"e7d49b95-af26-4035-9c96-7a362deb5bf5\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " Sep 15 21:34:00.620646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.620459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d49b95-af26-4035-9c96-7a362deb5bf5-extensions-socket-volume\") pod \"e7d49b95-af26-4035-9c96-7a362deb5bf5\" (UID: \"e7d49b95-af26-4035-9c96-7a362deb5bf5\") " Sep 15 21:34:00.620646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.620541 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2lch\" (UniqueName: \"kubernetes.io/projected/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1-kube-api-access-l2lch\") pod \"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1\" (UID: \"7f73eb91-27d0-4873-aa06-e7a5cf98c8b1\") " Sep 15 21:34:00.620876 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.620840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d49b95-af26-4035-9c96-7a362deb5bf5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7d49b95-af26-4035-9c96-7a362deb5bf5" (UID: "e7d49b95-af26-4035-9c96-7a362deb5bf5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:00.622925 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.622898 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d49b95-af26-4035-9c96-7a362deb5bf5-kube-api-access-b72t2" (OuterVolumeSpecName: "kube-api-access-b72t2") pod "e7d49b95-af26-4035-9c96-7a362deb5bf5" (UID: "e7d49b95-af26-4035-9c96-7a362deb5bf5"). InnerVolumeSpecName "kube-api-access-b72t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:00.623043 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.622947 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1-kube-api-access-l2lch" (OuterVolumeSpecName: "kube-api-access-l2lch") pod "7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" (UID: "7f73eb91-27d0-4873-aa06-e7a5cf98c8b1"). InnerVolumeSpecName "kube-api-access-l2lch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:00.633556 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.632111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:00.638951 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.638927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:00.721669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.721637 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2lch\" (UniqueName: \"kubernetes.io/projected/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1-kube-api-access-l2lch\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:00.721669 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.721666 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b72t2\" (UniqueName: \"kubernetes.io/projected/e7d49b95-af26-4035-9c96-7a362deb5bf5-kube-api-access-b72t2\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:00.722043 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.721677 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d49b95-af26-4035-9c96-7a362deb5bf5-extensions-socket-volume\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:00.920598 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.920566 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl"] Sep 15 21:34:00.921869 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:00.921849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c"] Sep 15 21:34:00.923778 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:34:00.923753 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32498c46_5964_4d1d_8f30_5aac8b7245f6.slice/crio-15a7e61e78af2aec6863a502f4d18385c9aec1dc505a341428fe8813ca02d701 WatchSource:0}: Error finding container 15a7e61e78af2aec6863a502f4d18385c9aec1dc505a341428fe8813ca02d701: Status 404 returned error can't find the container with id 15a7e61e78af2aec6863a502f4d18385c9aec1dc505a341428fe8813ca02d701 Sep 15 21:34:00.924768 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:34:00.924744 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffc2c65_9f5c_4877_ae27_1d5b17b5dfe6.slice/crio-771af2f8b4b9caac53815e05432ea2e6def79ee820393ec1af6eba4e077c1356 WatchSource:0}: Error finding container 771af2f8b4b9caac53815e05432ea2e6def79ee820393ec1af6eba4e077c1356: Status 404 returned error can't find the container with id 771af2f8b4b9caac53815e05432ea2e6def79ee820393ec1af6eba4e077c1356 Sep 15 21:34:01.073747 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.073719 2575 generic.go:356] "Generic (PLEG): container finished" podID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" containerID="0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466" exitCode=0 Sep 15 21:34:01.073916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.073764 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:01.073916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.073800 2575 scope.go:117] "RemoveContainer" containerID="0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466" Sep 15 21:34:01.075607 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.075576 2575 generic.go:356] "Generic (PLEG): container finished" podID="e7d49b95-af26-4035-9c96-7a362deb5bf5" containerID="125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a" exitCode=0 Sep 15 21:34:01.075750 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.075733 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:01.076850 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.076820 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:01.078642 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.078559 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:01.080013 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.079886 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" event={"ID":"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6","Type":"ContainerStarted","Data":"d43ba6fa58c997fd3c0f653aae9353a77cabc2bb952b4c7b8d95b5eb36d99398"} Sep 15 21:34:01.080013 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.079936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" event={"ID":"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6","Type":"ContainerStarted","Data":"771af2f8b4b9caac53815e05432ea2e6def79ee820393ec1af6eba4e077c1356"} Sep 15 21:34:01.080013 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.079983 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:01.080637 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.080509 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:01.082357 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.082327 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:01.082465 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.082403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:34:01.082523 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.082465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" event={"ID":"32498c46-5964-4d1d-8f30-5aac8b7245f6","Type":"ContainerStarted","Data":"dfecce338578ff0e06f9a72e79c5fd3576c0dfcefd46cf40cda8dcaf2db440ec"} Sep 15 21:34:01.082523 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.082504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" event={"ID":"32498c46-5964-4d1d-8f30-5aac8b7245f6","Type":"ContainerStarted","Data":"15a7e61e78af2aec6863a502f4d18385c9aec1dc505a341428fe8813ca02d701"} Sep 15 21:34:01.082765 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.082746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:01.084203 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.084175 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:01.086079 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.086058 2575 scope.go:117] "RemoveContainer" containerID="0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466" Sep 15 21:34:01.086447 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.086429 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466\": container with ID starting with 0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466 not found: ID does not exist" containerID="0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466" Sep 15 21:34:01.086549 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.086455 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466"} err="failed to get container status \"0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466\": rpc error: code = NotFound desc = could not find container \"0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466\": container with ID starting with 0d2348835f55c1080c2b4b6a8bdfd5ada61baa5ee2c06da33882d1e37431e466 not found: ID does not exist" Sep 15 21:34:01.086549 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.086471 2575 scope.go:117] "RemoveContainer" containerID="125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a" Sep 15 21:34:01.095244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.095219 2575 scope.go:117] "RemoveContainer" containerID="125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a" Sep 15 21:34:01.095546 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.095529 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a\": container with ID starting with 125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a not found: ID does not exist" containerID="125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a" Sep 15 21:34:01.095590 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.095554 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a"} err="failed to get container status \"125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a\": rpc error: code = NotFound desc = could not find container \"125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a\": container with ID starting with 125409351c1b5835555f32a12214f7d7e8d1f06101ffa57ed61bfbab1c78d96a not found: ID does not exist" Sep 15 21:34:01.104796 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.104756 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" podStartSLOduration=1.104743199 podStartE2EDuration="1.104743199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:01.103961882 +0000 UTC m=+490.443441347" watchObservedRunningTime="2026-09-15 21:34:01.104743199 +0000 UTC m=+490.444222650" Sep 15 21:34:01.105954 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.105934 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:01.107714 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.107696 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-l6zlc\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-l6zlc" Sep 15 21:34:01.109202 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.109185 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-cq7xp\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7057d956-e6b7-4acb-bbc0-bd9c7d874524" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cq7xp" Sep 15 21:34:01.146512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.146485 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7057d956-e6b7-4acb-bbc0-bd9c7d874524" path="/var/lib/kubelet/pods/7057d956-e6b7-4acb-bbc0-bd9c7d874524/volumes" Sep 15 21:34:01.146697 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.146685 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" path="/var/lib/kubelet/pods/7f73eb91-27d0-4873-aa06-e7a5cf98c8b1/volumes" Sep 15 21:34:01.146980 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.146969 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d49b95-af26-4035-9c96-7a362deb5bf5" path="/var/lib/kubelet/pods/e7d49b95-af26-4035-9c96-7a362deb5bf5/volumes" Sep 15 21:34:01.153086 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:01.153044 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" podStartSLOduration=1.153031489 podStartE2EDuration="1.153031489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:01.1520025 +0000 UTC m=+490.491481951" watchObservedRunningTime="2026-09-15 21:34:01.153031489 +0000 UTC m=+490.492510940" Sep 15 21:34:01.154692 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:01.154671 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zpg79\" is forbidden: User \"system:node:ip-10-0-130-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-52.ec2.internal' and this object" podUID="7f73eb91-27d0-4873-aa06-e7a5cf98c8b1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zpg79" Sep 15 21:34:12.091509 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:12.091481 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:12.091965 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:12.091578 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kch8c" Sep 15 21:34:24.034687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.034653 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl"] Sep 15 21:34:24.035061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.034848 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" podUID="dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" containerName="manager" containerID="cri-o://d43ba6fa58c997fd3c0f653aae9353a77cabc2bb952b4c7b8d95b5eb36d99398" gracePeriod=10 Sep 15 21:34:24.189188 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.189151 2575 generic.go:356] "Generic (PLEG): container finished" podID="dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" containerID="d43ba6fa58c997fd3c0f653aae9353a77cabc2bb952b4c7b8d95b5eb36d99398" exitCode=0 Sep 15 21:34:24.189337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.189226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" event={"ID":"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6","Type":"ContainerDied","Data":"d43ba6fa58c997fd3c0f653aae9353a77cabc2bb952b4c7b8d95b5eb36d99398"} Sep 15 21:34:24.281507 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.281486 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:24.298380 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.298312 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z"] Sep 15 21:34:24.298666 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.298653 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" containerName="manager" Sep 15 21:34:24.298706 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.298668 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" containerName="manager" Sep 15 21:34:24.298738 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.298722 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" containerName="manager" Sep 15 21:34:24.300599 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.300584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.328529 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.328498 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z"] Sep 15 21:34:24.400853 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.400804 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-extensions-socket-volume\") pod \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " Sep 15 21:34:24.400853 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.400854 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4qnl\" (UniqueName: \"kubernetes.io/projected/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-kube-api-access-g4qnl\") pod \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\" (UID: \"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6\") " Sep 15 21:34:24.401135 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.401017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8vm\" (UniqueName: \"kubernetes.io/projected/2020d892-df37-466b-8134-3093e91d366b-kube-api-access-rf8vm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9qw2z\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.401135 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.401087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020d892-df37-466b-8134-3093e91d366b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9qw2z\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.401223 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.401204 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" (UID: "dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:34:24.403083 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.403058 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-kube-api-access-g4qnl" (OuterVolumeSpecName: "kube-api-access-g4qnl") pod "dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" (UID: "dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6"). InnerVolumeSpecName "kube-api-access-g4qnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:24.502285 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.502227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8vm\" (UniqueName: \"kubernetes.io/projected/2020d892-df37-466b-8134-3093e91d366b-kube-api-access-rf8vm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9qw2z\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.502452 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.502303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020d892-df37-466b-8134-3093e91d366b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9qw2z\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.502452 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.502334 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-extensions-socket-volume\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:24.502452 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.502344 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qnl\" (UniqueName: \"kubernetes.io/projected/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6-kube-api-access-g4qnl\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:24.502634 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.502614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020d892-df37-466b-8134-3093e91d366b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9qw2z\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.510938 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.510905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8vm\" (UniqueName: \"kubernetes.io/projected/2020d892-df37-466b-8134-3093e91d366b-kube-api-access-rf8vm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9qw2z\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.611859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.611770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:24.890711 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:24.890632 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z"] Sep 15 21:34:24.894288 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:34:24.894264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2020d892_df37_466b_8134_3093e91d366b.slice/crio-a97cc4f4e7dc90216d928ba460ae0329ee3bcfb6022bc1d1f0cff9496921dcdb WatchSource:0}: Error finding container a97cc4f4e7dc90216d928ba460ae0329ee3bcfb6022bc1d1f0cff9496921dcdb: Status 404 returned error can't find the container with id a97cc4f4e7dc90216d928ba460ae0329ee3bcfb6022bc1d1f0cff9496921dcdb Sep 15 21:34:25.195463 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.195432 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" Sep 15 21:34:25.195867 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.195437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl" event={"ID":"dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6","Type":"ContainerDied","Data":"771af2f8b4b9caac53815e05432ea2e6def79ee820393ec1af6eba4e077c1356"} Sep 15 21:34:25.195867 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.195581 2575 scope.go:117] "RemoveContainer" containerID="d43ba6fa58c997fd3c0f653aae9353a77cabc2bb952b4c7b8d95b5eb36d99398" Sep 15 21:34:25.197591 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.197561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" event={"ID":"2020d892-df37-466b-8134-3093e91d366b","Type":"ContainerStarted","Data":"ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4"} Sep 15 21:34:25.197719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.197619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" event={"ID":"2020d892-df37-466b-8134-3093e91d366b","Type":"ContainerStarted","Data":"a97cc4f4e7dc90216d928ba460ae0329ee3bcfb6022bc1d1f0cff9496921dcdb"} Sep 15 21:34:25.197880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.197860 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:25.216617 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.216563 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" podStartSLOduration=1.2165431820000001 podStartE2EDuration="1.216543182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:25.215580664 +0000 UTC m=+514.555060116" watchObservedRunningTime="2026-09-15 21:34:25.216543182 +0000 UTC m=+514.556022634" Sep 15 21:34:25.229606 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.229572 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl"] Sep 15 21:34:25.235586 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:25.235560 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sttkl"] Sep 15 21:34:27.147513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:27.147485 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6" path="/var/lib/kubelet/pods/dffc2c65-9f5c-4877-ae27-1d5b17b5dfe6/volumes" Sep 15 21:34:36.205815 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:36.205788 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:34:43.336243 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.336153 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:34:43.339253 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.339237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.341605 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.341574 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 15 21:34:43.341720 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.341667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-csqbq\"" Sep 15 21:34:43.346974 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.346951 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:34:43.351086 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.351061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f7850a1e-695d-4f6c-b1d7-420e237c35cb-config-file\") pod \"limitador-limitador-8588749977-d7759\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.351214 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.351188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zf8v\" (UniqueName: \"kubernetes.io/projected/f7850a1e-695d-4f6c-b1d7-420e237c35cb-kube-api-access-4zf8v\") pod \"limitador-limitador-8588749977-d7759\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.436832 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.436792 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:34:43.451533 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.451505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zf8v\" (UniqueName: \"kubernetes.io/projected/f7850a1e-695d-4f6c-b1d7-420e237c35cb-kube-api-access-4zf8v\") pod \"limitador-limitador-8588749977-d7759\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.451720 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.451600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f7850a1e-695d-4f6c-b1d7-420e237c35cb-config-file\") pod \"limitador-limitador-8588749977-d7759\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.452341 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.452317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f7850a1e-695d-4f6c-b1d7-420e237c35cb-config-file\") pod \"limitador-limitador-8588749977-d7759\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.459177 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.459157 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zf8v\" (UniqueName: \"kubernetes.io/projected/f7850a1e-695d-4f6c-b1d7-420e237c35cb-kube-api-access-4zf8v\") pod \"limitador-limitador-8588749977-d7759\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.650753 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.650672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:43.918602 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:43.918570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:34:43.920729 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:34:43.920703 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7850a1e_695d_4f6c_b1d7_420e237c35cb.slice/crio-418d0d52371987248720fcf5cffad13bc44ae5274c534fb3f7e7fd7ad56f7c60 WatchSource:0}: Error finding container 418d0d52371987248720fcf5cffad13bc44ae5274c534fb3f7e7fd7ad56f7c60: Status 404 returned error can't find the container with id 418d0d52371987248720fcf5cffad13bc44ae5274c534fb3f7e7fd7ad56f7c60 Sep 15 21:34:44.132310 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.132276 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65b82"] Sep 15 21:34:44.136778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.136751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:44.138963 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.138943 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2lqt6\"" Sep 15 21:34:44.141443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.141424 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65b82"] Sep 15 21:34:44.156953 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.156921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47gf6\" (UniqueName: \"kubernetes.io/projected/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa-kube-api-access-47gf6\") pod \"authorino-7bcfd8f9cd-65b82\" (UID: \"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa\") " pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:44.257415 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.257354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47gf6\" (UniqueName: \"kubernetes.io/projected/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa-kube-api-access-47gf6\") pod \"authorino-7bcfd8f9cd-65b82\" (UID: \"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa\") " pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:44.270930 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.270909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47gf6\" (UniqueName: \"kubernetes.io/projected/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa-kube-api-access-47gf6\") pod \"authorino-7bcfd8f9cd-65b82\" (UID: \"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa\") " pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:44.280622 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.280600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d7759" event={"ID":"f7850a1e-695d-4f6c-b1d7-420e237c35cb","Type":"ContainerStarted","Data":"418d0d52371987248720fcf5cffad13bc44ae5274c534fb3f7e7fd7ad56f7c60"} Sep 15 21:34:44.398862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.398828 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-fb8s5"] Sep 15 21:34:44.402144 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.402123 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:34:44.406344 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.406316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fb8s5"] Sep 15 21:34:44.447333 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.447304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:44.458673 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.458648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvpt\" (UniqueName: \"kubernetes.io/projected/782a35e0-3bca-4ae8-b68c-5b4edcdd1811-kube-api-access-8lvpt\") pod \"authorino-59c974bd48-fb8s5\" (UID: \"782a35e0-3bca-4ae8-b68c-5b4edcdd1811\") " pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:34:44.559557 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.559479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvpt\" (UniqueName: \"kubernetes.io/projected/782a35e0-3bca-4ae8-b68c-5b4edcdd1811-kube-api-access-8lvpt\") pod \"authorino-59c974bd48-fb8s5\" (UID: \"782a35e0-3bca-4ae8-b68c-5b4edcdd1811\") " pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:34:44.567887 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.567867 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvpt\" (UniqueName: \"kubernetes.io/projected/782a35e0-3bca-4ae8-b68c-5b4edcdd1811-kube-api-access-8lvpt\") pod \"authorino-59c974bd48-fb8s5\" (UID: \"782a35e0-3bca-4ae8-b68c-5b4edcdd1811\") " pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:34:44.712894 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.712864 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:34:44.718550 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.718512 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65b82"] Sep 15 21:34:44.722939 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:34:44.722909 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70e1ee40_3f13_40ed_aac0_2e1cea4d1caa.slice/crio-96061eee22f7d6956a200b16b4b5231111bb6b3006006611acc9d64a46033c6f WatchSource:0}: Error finding container 96061eee22f7d6956a200b16b4b5231111bb6b3006006611acc9d64a46033c6f: Status 404 returned error can't find the container with id 96061eee22f7d6956a200b16b4b5231111bb6b3006006611acc9d64a46033c6f Sep 15 21:34:44.985482 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:44.985449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fb8s5"] Sep 15 21:34:45.157535 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:34:45.157503 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod782a35e0_3bca_4ae8_b68c_5b4edcdd1811.slice/crio-0d9e2c4e4cee8321ae3bcd621ae30a200ff546b0a3553e8ca427b032b278e349 WatchSource:0}: Error finding container 0d9e2c4e4cee8321ae3bcd621ae30a200ff546b0a3553e8ca427b032b278e349: Status 404 returned error can't find the container with id 0d9e2c4e4cee8321ae3bcd621ae30a200ff546b0a3553e8ca427b032b278e349 Sep 15 21:34:45.287847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:45.287807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d7759" event={"ID":"f7850a1e-695d-4f6c-b1d7-420e237c35cb","Type":"ContainerStarted","Data":"1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0"} Sep 15 21:34:45.288048 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:45.287925 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:34:45.289451 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:45.289424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fb8s5" event={"ID":"782a35e0-3bca-4ae8-b68c-5b4edcdd1811","Type":"ContainerStarted","Data":"0d9e2c4e4cee8321ae3bcd621ae30a200ff546b0a3553e8ca427b032b278e349"} Sep 15 21:34:45.290594 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:45.290575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" event={"ID":"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa","Type":"ContainerStarted","Data":"96061eee22f7d6956a200b16b4b5231111bb6b3006006611acc9d64a46033c6f"} Sep 15 21:34:45.303526 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:45.303485 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-d7759" podStartSLOduration=1.014860955 podStartE2EDuration="2.303472261s" totalImagesPullingTime="1.288611306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:45.302641749 +0000 UTC m=+534.642121201" watchObservedRunningTime="2026-09-15 21:34:45.303472261 +0000 UTC m=+534.642951711" Sep 15 21:34:47.303307 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:47.303264 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fb8s5" event={"ID":"782a35e0-3bca-4ae8-b68c-5b4edcdd1811","Type":"ContainerStarted","Data":"f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2"} Sep 15 21:34:47.304843 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:47.304818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" event={"ID":"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa","Type":"ContainerStarted","Data":"0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5"} Sep 15 21:34:47.319107 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:47.319049 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-fb8s5" podStartSLOduration=1.904653584 podStartE2EDuration="3.319034258s" totalImagesPullingTime="1.414380674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:47.318177779 +0000 UTC m=+536.657657231" watchObservedRunningTime="2026-09-15 21:34:47.319034258 +0000 UTC m=+536.658513709" Sep 15 21:34:47.332154 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:47.332083 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" podStartSLOduration=1.486978383 podStartE2EDuration="3.33206417s" totalImagesPullingTime="1.845085787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:47.3317775 +0000 UTC m=+536.671256952" watchObservedRunningTime="2026-09-15 21:34:47.33206417 +0000 UTC m=+536.671543621" Sep 15 21:34:47.378330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:47.378300 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65b82"] Sep 15 21:34:49.314111 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:49.314071 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" podUID="70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" containerName="authorino" containerID="cri-o://0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5" gracePeriod=30 Sep 15 21:34:49.555727 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:49.555706 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:49.597831 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:49.597756 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47gf6\" (UniqueName: \"kubernetes.io/projected/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa-kube-api-access-47gf6\") pod \"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa\" (UID: \"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa\") " Sep 15 21:34:49.599865 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:49.599834 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa-kube-api-access-47gf6" (OuterVolumeSpecName: "kube-api-access-47gf6") pod "70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" (UID: "70e1ee40-3f13-40ed-aac0-2e1cea4d1caa"). InnerVolumeSpecName "kube-api-access-47gf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:34:49.698273 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:49.698244 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47gf6\" (UniqueName: \"kubernetes.io/projected/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa-kube-api-access-47gf6\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:34:50.320700 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.320663 2575 generic.go:356] "Generic (PLEG): container finished" podID="70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" containerID="0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5" exitCode=0 Sep 15 21:34:50.321160 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.320722 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" Sep 15 21:34:50.321160 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.320747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" event={"ID":"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa","Type":"ContainerDied","Data":"0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5"} Sep 15 21:34:50.321160 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.320791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65b82" event={"ID":"70e1ee40-3f13-40ed-aac0-2e1cea4d1caa","Type":"ContainerDied","Data":"96061eee22f7d6956a200b16b4b5231111bb6b3006006611acc9d64a46033c6f"} Sep 15 21:34:50.321160 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.320807 2575 scope.go:117] "RemoveContainer" containerID="0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5" Sep 15 21:34:50.330276 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.330256 2575 scope.go:117] "RemoveContainer" containerID="0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5" Sep 15 21:34:50.330573 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:34:50.330546 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5\": container with ID starting with 0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5 not found: ID does not exist" containerID="0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5" Sep 15 21:34:50.330674 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.330584 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5"} err="failed to get container status \"0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5\": rpc error: code = NotFound desc = could not find container \"0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5\": container with ID starting with 0a64d7c535aed3b9a071b700847664d6f2d606b3086b1e937659b827353182e5 not found: ID does not exist" Sep 15 21:34:50.340472 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.340452 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65b82"] Sep 15 21:34:50.343921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:50.343887 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65b82"] Sep 15 21:34:51.147230 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:51.147196 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" path="/var/lib/kubelet/pods/70e1ee40-3f13-40ed-aac0-2e1cea4d1caa/volumes" Sep 15 21:34:56.298905 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:34:56.298869 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:35:09.251272 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.251234 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:35:09.251909 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.251473 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-d7759" podUID="f7850a1e-695d-4f6c-b1d7-420e237c35cb" containerName="limitador" containerID="cri-o://1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0" gracePeriod=30 Sep 15 21:35:09.800545 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.800519 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:35:09.858430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.858358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f7850a1e-695d-4f6c-b1d7-420e237c35cb-config-file\") pod \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " Sep 15 21:35:09.858574 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.858435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zf8v\" (UniqueName: \"kubernetes.io/projected/f7850a1e-695d-4f6c-b1d7-420e237c35cb-kube-api-access-4zf8v\") pod \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\" (UID: \"f7850a1e-695d-4f6c-b1d7-420e237c35cb\") " Sep 15 21:35:09.858713 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.858690 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7850a1e-695d-4f6c-b1d7-420e237c35cb-config-file" (OuterVolumeSpecName: "config-file") pod "f7850a1e-695d-4f6c-b1d7-420e237c35cb" (UID: "f7850a1e-695d-4f6c-b1d7-420e237c35cb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:35:09.860570 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.860543 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7850a1e-695d-4f6c-b1d7-420e237c35cb-kube-api-access-4zf8v" (OuterVolumeSpecName: "kube-api-access-4zf8v") pod "f7850a1e-695d-4f6c-b1d7-420e237c35cb" (UID: "f7850a1e-695d-4f6c-b1d7-420e237c35cb"). InnerVolumeSpecName "kube-api-access-4zf8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:35:09.959261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.959227 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zf8v\" (UniqueName: \"kubernetes.io/projected/f7850a1e-695d-4f6c-b1d7-420e237c35cb-kube-api-access-4zf8v\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:09.959261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:09.959255 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f7850a1e-695d-4f6c-b1d7-420e237c35cb-config-file\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:10.407219 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.407181 2575 generic.go:356] "Generic (PLEG): container finished" podID="f7850a1e-695d-4f6c-b1d7-420e237c35cb" containerID="1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0" exitCode=0 Sep 15 21:35:10.407650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.407228 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-d7759" Sep 15 21:35:10.407650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.407270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d7759" event={"ID":"f7850a1e-695d-4f6c-b1d7-420e237c35cb","Type":"ContainerDied","Data":"1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0"} Sep 15 21:35:10.407650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.407334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-d7759" event={"ID":"f7850a1e-695d-4f6c-b1d7-420e237c35cb","Type":"ContainerDied","Data":"418d0d52371987248720fcf5cffad13bc44ae5274c534fb3f7e7fd7ad56f7c60"} Sep 15 21:35:10.407650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.407361 2575 scope.go:117] "RemoveContainer" containerID="1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0" Sep 15 21:35:10.418486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.418466 2575 scope.go:117] "RemoveContainer" containerID="1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0" Sep 15 21:35:10.418981 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:35:10.418956 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0\": container with ID starting with 1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0 not found: ID does not exist" containerID="1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0" Sep 15 21:35:10.419075 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.418992 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0"} err="failed to get container status \"1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0\": rpc error: code = NotFound desc = could not find container \"1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0\": container with ID starting with 1096cd4ffba6c9e60b4f0bc4e8f62562c5e6fe8c85f69664828b012b37a7c9b0 not found: ID does not exist" Sep 15 21:35:10.428760 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.428738 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:35:10.432518 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:10.432497 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-d7759"] Sep 15 21:35:11.146859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:11.146822 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7850a1e-695d-4f6c-b1d7-420e237c35cb" path="/var/lib/kubelet/pods/f7850a1e-695d-4f6c-b1d7-420e237c35cb/volumes" Sep 15 21:35:14.353668 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.353633 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mc522"] Sep 15 21:35:14.354053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.353966 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" containerName="authorino" Sep 15 21:35:14.354053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.353981 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" containerName="authorino" Sep 15 21:35:14.354053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.354003 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7850a1e-695d-4f6c-b1d7-420e237c35cb" containerName="limitador" Sep 15 21:35:14.354053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.354008 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7850a1e-695d-4f6c-b1d7-420e237c35cb" containerName="limitador" Sep 15 21:35:14.354225 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.354065 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7850a1e-695d-4f6c-b1d7-420e237c35cb" containerName="limitador" Sep 15 21:35:14.354225 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.354075 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e1ee40-3f13-40ed-aac0-2e1cea4d1caa" containerName="authorino" Sep 15 21:35:14.356855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.356839 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.359701 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.359662 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tl9hh\"" Sep 15 21:35:14.359811 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.359712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 21:35:14.360706 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.360673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 15 21:35:14.360924 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.360862 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 21:35:14.365984 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.365964 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mc522"] Sep 15 21:35:14.497314 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.497283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2541a-85b3-4899-bf60-4f0c70474ba5-data\") pod \"postgres-69fb4594cf-mc522\" (UID: \"b4a2541a-85b3-4899-bf60-4f0c70474ba5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.497494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.497336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l29s\" (UniqueName: \"kubernetes.io/projected/b4a2541a-85b3-4899-bf60-4f0c70474ba5-kube-api-access-8l29s\") pod \"postgres-69fb4594cf-mc522\" (UID: \"b4a2541a-85b3-4899-bf60-4f0c70474ba5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.598561 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.598515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l29s\" (UniqueName: \"kubernetes.io/projected/b4a2541a-85b3-4899-bf60-4f0c70474ba5-kube-api-access-8l29s\") pod \"postgres-69fb4594cf-mc522\" (UID: \"b4a2541a-85b3-4899-bf60-4f0c70474ba5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.598803 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.598605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2541a-85b3-4899-bf60-4f0c70474ba5-data\") pod \"postgres-69fb4594cf-mc522\" (UID: \"b4a2541a-85b3-4899-bf60-4f0c70474ba5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.598919 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.598904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2541a-85b3-4899-bf60-4f0c70474ba5-data\") pod \"postgres-69fb4594cf-mc522\" (UID: \"b4a2541a-85b3-4899-bf60-4f0c70474ba5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.606811 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.606757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l29s\" (UniqueName: \"kubernetes.io/projected/b4a2541a-85b3-4899-bf60-4f0c70474ba5-kube-api-access-8l29s\") pod \"postgres-69fb4594cf-mc522\" (UID: \"b4a2541a-85b3-4899-bf60-4f0c70474ba5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.667482 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.667459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:14.935868 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:14.935702 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mc522"] Sep 15 21:35:14.938531 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:35:14.938501 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a2541a_85b3_4899_bf60_4f0c70474ba5.slice/crio-a5a0f751974cd45d2f1510cc19a823b64290d45d687111e16575efdf4483163a WatchSource:0}: Error finding container a5a0f751974cd45d2f1510cc19a823b64290d45d687111e16575efdf4483163a: Status 404 returned error can't find the container with id a5a0f751974cd45d2f1510cc19a823b64290d45d687111e16575efdf4483163a Sep 15 21:35:15.432465 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:15.432424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" event={"ID":"b4a2541a-85b3-4899-bf60-4f0c70474ba5","Type":"ContainerStarted","Data":"a5a0f751974cd45d2f1510cc19a823b64290d45d687111e16575efdf4483163a"} Sep 15 21:35:20.462166 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:20.462129 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" event={"ID":"b4a2541a-85b3-4899-bf60-4f0c70474ba5","Type":"ContainerStarted","Data":"ea41f76b776e037c0d9d06d021d505098213e5ecc804cabc7e4fa8c70971b062"} Sep 15 21:35:20.462548 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:20.462196 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:20.478828 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:20.478781 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" podStartSLOduration=1.406057905 podStartE2EDuration="6.478765451s" totalImagesPullingTime="5.072707546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:20.47718615 +0000 UTC m=+569.816665602" watchObservedRunningTime="2026-09-15 21:35:20.478765451 +0000 UTC m=+569.818244901" Sep 15 21:35:26.497854 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:26.497822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mc522" Sep 15 21:35:27.047829 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.047788 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-rvd4n"] Sep 15 21:35:27.051418 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.051394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:27.057629 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.057601 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-rvd4n"] Sep 15 21:35:27.162554 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.162516 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rvd4n"] Sep 15 21:35:27.162776 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:35:27.162757 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-h6nvb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-rvd4n" podUID="68c5542d-e721-4ed1-8fd2-e69e74ff67c6" Sep 15 21:35:27.180461 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.180430 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qgfsn"] Sep 15 21:35:27.183668 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.183653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.188287 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.188264 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 15 21:35:27.193555 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.193533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qgfsn"] Sep 15 21:35:27.204164 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.204089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nvb\" (UniqueName: \"kubernetes.io/projected/68c5542d-e721-4ed1-8fd2-e69e74ff67c6-kube-api-access-h6nvb\") pod \"authorino-557f876f69-rvd4n\" (UID: \"68c5542d-e721-4ed1-8fd2-e69e74ff67c6\") " pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:27.288256 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.288224 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qgfsn"] Sep 15 21:35:27.288478 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:35:27.288460 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dg4rc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" podUID="3e5aa860-4751-4702-98f8-abc3bf9fe5b5" Sep 15 21:35:27.304700 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.304631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nvb\" (UniqueName: \"kubernetes.io/projected/68c5542d-e721-4ed1-8fd2-e69e74ff67c6-kube-api-access-h6nvb\") pod \"authorino-557f876f69-rvd4n\" (UID: \"68c5542d-e721-4ed1-8fd2-e69e74ff67c6\") " pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:27.304700 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.304670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-tls-cert\") pod \"authorino-6d4dfc649f-qgfsn\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.304896 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.304712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4rc\" (UniqueName: \"kubernetes.io/projected/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-kube-api-access-dg4rc\") pod \"authorino-6d4dfc649f-qgfsn\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.312677 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.312648 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nvb\" (UniqueName: \"kubernetes.io/projected/68c5542d-e721-4ed1-8fd2-e69e74ff67c6-kube-api-access-h6nvb\") pod \"authorino-557f876f69-rvd4n\" (UID: \"68c5542d-e721-4ed1-8fd2-e69e74ff67c6\") " pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:27.317627 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.317595 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dbcb84b8c-zrv8m"] Sep 15 21:35:27.321029 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.321010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.327613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.327591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dbcb84b8c-zrv8m"] Sep 15 21:35:27.406055 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.406026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4rc\" (UniqueName: \"kubernetes.io/projected/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-kube-api-access-dg4rc\") pod \"authorino-6d4dfc649f-qgfsn\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.406239 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.406061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed89d270-f7e8-470d-aba3-44772b708246-tls-cert\") pod \"authorino-7dbcb84b8c-zrv8m\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.406239 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.406128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6s7\" (UniqueName: \"kubernetes.io/projected/ed89d270-f7e8-470d-aba3-44772b708246-kube-api-access-gj6s7\") pod \"authorino-7dbcb84b8c-zrv8m\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.406239 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.406161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-tls-cert\") pod \"authorino-6d4dfc649f-qgfsn\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.408751 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.408729 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-tls-cert\") pod \"authorino-6d4dfc649f-qgfsn\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.414092 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.414073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4rc\" (UniqueName: \"kubernetes.io/projected/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-kube-api-access-dg4rc\") pod \"authorino-6d4dfc649f-qgfsn\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.494387 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.494359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.494571 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.494358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:27.501756 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.501737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:27.504780 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.504760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:27.507149 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.507127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed89d270-f7e8-470d-aba3-44772b708246-tls-cert\") pod \"authorino-7dbcb84b8c-zrv8m\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.507273 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.507250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6s7\" (UniqueName: \"kubernetes.io/projected/ed89d270-f7e8-470d-aba3-44772b708246-kube-api-access-gj6s7\") pod \"authorino-7dbcb84b8c-zrv8m\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.509835 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.509808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed89d270-f7e8-470d-aba3-44772b708246-tls-cert\") pod \"authorino-7dbcb84b8c-zrv8m\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.515077 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.515051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6s7\" (UniqueName: \"kubernetes.io/projected/ed89d270-f7e8-470d-aba3-44772b708246-kube-api-access-gj6s7\") pod \"authorino-7dbcb84b8c-zrv8m\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.608321 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.608220 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6nvb\" (UniqueName: \"kubernetes.io/projected/68c5542d-e721-4ed1-8fd2-e69e74ff67c6-kube-api-access-h6nvb\") pod \"68c5542d-e721-4ed1-8fd2-e69e74ff67c6\" (UID: \"68c5542d-e721-4ed1-8fd2-e69e74ff67c6\") " Sep 15 21:35:27.608321 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.608279 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg4rc\" (UniqueName: \"kubernetes.io/projected/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-kube-api-access-dg4rc\") pod \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " Sep 15 21:35:27.608546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.608334 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-tls-cert\") pod \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\" (UID: \"3e5aa860-4751-4702-98f8-abc3bf9fe5b5\") " Sep 15 21:35:27.610499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.610467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3e5aa860-4751-4702-98f8-abc3bf9fe5b5" (UID: "3e5aa860-4751-4702-98f8-abc3bf9fe5b5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:35:27.610615 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.610526 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c5542d-e721-4ed1-8fd2-e69e74ff67c6-kube-api-access-h6nvb" (OuterVolumeSpecName: "kube-api-access-h6nvb") pod "68c5542d-e721-4ed1-8fd2-e69e74ff67c6" (UID: "68c5542d-e721-4ed1-8fd2-e69e74ff67c6"). InnerVolumeSpecName "kube-api-access-h6nvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:35:27.610615 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.610535 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-kube-api-access-dg4rc" (OuterVolumeSpecName: "kube-api-access-dg4rc") pod "3e5aa860-4751-4702-98f8-abc3bf9fe5b5" (UID: "3e5aa860-4751-4702-98f8-abc3bf9fe5b5"). InnerVolumeSpecName "kube-api-access-dg4rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:35:27.631280 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.631254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:35:27.711196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.709506 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4rc\" (UniqueName: \"kubernetes.io/projected/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-kube-api-access-dg4rc\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:27.711196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.709547 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e5aa860-4751-4702-98f8-abc3bf9fe5b5-tls-cert\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:27.711196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.709563 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6nvb\" (UniqueName: \"kubernetes.io/projected/68c5542d-e721-4ed1-8fd2-e69e74ff67c6-kube-api-access-h6nvb\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:27.901620 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:27.901597 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dbcb84b8c-zrv8m"] Sep 15 21:35:27.904001 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:35:27.903973 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded89d270_f7e8_470d_aba3_44772b708246.slice/crio-a893e08db56bc9a4aca9d9adb24208b464f4d277f00854f433eaded303c99cea WatchSource:0}: Error finding container a893e08db56bc9a4aca9d9adb24208b464f4d277f00854f433eaded303c99cea: Status 404 returned error can't find the container with id a893e08db56bc9a4aca9d9adb24208b464f4d277f00854f433eaded303c99cea Sep 15 21:35:28.500073 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.500033 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" event={"ID":"ed89d270-f7e8-470d-aba3-44772b708246","Type":"ContainerStarted","Data":"a893e08db56bc9a4aca9d9adb24208b464f4d277f00854f433eaded303c99cea"} Sep 15 21:35:28.500073 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.500068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qgfsn" Sep 15 21:35:28.500310 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.500080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rvd4n" Sep 15 21:35:28.530318 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.530285 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rvd4n"] Sep 15 21:35:28.532289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.532265 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-rvd4n"] Sep 15 21:35:28.552496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.552462 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qgfsn"] Sep 15 21:35:28.554797 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:28.554774 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qgfsn"] Sep 15 21:35:29.147154 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.147055 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e5aa860-4751-4702-98f8-abc3bf9fe5b5" path="/var/lib/kubelet/pods/3e5aa860-4751-4702-98f8-abc3bf9fe5b5/volumes" Sep 15 21:35:29.147298 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.147288 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c5542d-e721-4ed1-8fd2-e69e74ff67c6" path="/var/lib/kubelet/pods/68c5542d-e721-4ed1-8fd2-e69e74ff67c6/volumes" Sep 15 21:35:29.506667 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.506635 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" event={"ID":"ed89d270-f7e8-470d-aba3-44772b708246","Type":"ContainerStarted","Data":"cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e"} Sep 15 21:35:29.522041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.521998 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" podStartSLOduration=1.604546598 podStartE2EDuration="2.521983215s" totalImagesPullingTime="917.436617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:29.520736887 +0000 UTC m=+578.860216339" watchObservedRunningTime="2026-09-15 21:35:29.521983215 +0000 UTC m=+578.861462665" Sep 15 21:35:29.545947 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.545871 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fb8s5"] Sep 15 21:35:29.546406 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.546162 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-fb8s5" podUID="782a35e0-3bca-4ae8-b68c-5b4edcdd1811" containerName="authorino" containerID="cri-o://f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2" gracePeriod=30 Sep 15 21:35:29.792936 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.792912 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:35:29.928838 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.928807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lvpt\" (UniqueName: \"kubernetes.io/projected/782a35e0-3bca-4ae8-b68c-5b4edcdd1811-kube-api-access-8lvpt\") pod \"782a35e0-3bca-4ae8-b68c-5b4edcdd1811\" (UID: \"782a35e0-3bca-4ae8-b68c-5b4edcdd1811\") " Sep 15 21:35:29.930990 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:29.930966 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782a35e0-3bca-4ae8-b68c-5b4edcdd1811-kube-api-access-8lvpt" (OuterVolumeSpecName: "kube-api-access-8lvpt") pod "782a35e0-3bca-4ae8-b68c-5b4edcdd1811" (UID: "782a35e0-3bca-4ae8-b68c-5b4edcdd1811"). InnerVolumeSpecName "kube-api-access-8lvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:35:30.029526 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.029453 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvpt\" (UniqueName: \"kubernetes.io/projected/782a35e0-3bca-4ae8-b68c-5b4edcdd1811-kube-api-access-8lvpt\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:30.512726 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.512696 2575 generic.go:356] "Generic (PLEG): container finished" podID="782a35e0-3bca-4ae8-b68c-5b4edcdd1811" containerID="f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2" exitCode=0 Sep 15 21:35:30.512916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.512741 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fb8s5" Sep 15 21:35:30.512916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.512784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fb8s5" event={"ID":"782a35e0-3bca-4ae8-b68c-5b4edcdd1811","Type":"ContainerDied","Data":"f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2"} Sep 15 21:35:30.512916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.512821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fb8s5" event={"ID":"782a35e0-3bca-4ae8-b68c-5b4edcdd1811","Type":"ContainerDied","Data":"0d9e2c4e4cee8321ae3bcd621ae30a200ff546b0a3553e8ca427b032b278e349"} Sep 15 21:35:30.512916 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.512848 2575 scope.go:117] "RemoveContainer" containerID="f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2" Sep 15 21:35:30.521968 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.521952 2575 scope.go:117] "RemoveContainer" containerID="f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2" Sep 15 21:35:30.522282 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:35:30.522260 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2\": container with ID starting with f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2 not found: ID does not exist" containerID="f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2" Sep 15 21:35:30.522409 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.522288 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2"} err="failed to get container status \"f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2\": rpc error: code = NotFound desc = could not find container \"f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2\": container with ID starting with f767e2753f9350ffa5d9e54165a83b29f540605e1dfbe55c3062f7d882ee09e2 not found: ID does not exist" Sep 15 21:35:30.533093 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.533068 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fb8s5"] Sep 15 21:35:30.537433 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:30.537414 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fb8s5"] Sep 15 21:35:31.146633 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:31.146605 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="782a35e0-3bca-4ae8-b68c-5b4edcdd1811" path="/var/lib/kubelet/pods/782a35e0-3bca-4ae8-b68c-5b4edcdd1811/volumes" Sep 15 21:35:34.238000 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.237971 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-98cf4f94d-ccg4v"] Sep 15 21:35:34.238493 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.238466 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782a35e0-3bca-4ae8-b68c-5b4edcdd1811" containerName="authorino" Sep 15 21:35:34.238493 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.238485 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="782a35e0-3bca-4ae8-b68c-5b4edcdd1811" containerName="authorino" Sep 15 21:35:34.238604 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.238579 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="782a35e0-3bca-4ae8-b68c-5b4edcdd1811" containerName="authorino" Sep 15 21:35:34.242782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.242762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.245223 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.245199 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 21:35:34.245443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.245230 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 21:35:34.245443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.245263 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 21:35:34.245602 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.245271 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svx2b\"" Sep 15 21:35:34.249014 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.248991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-98cf4f94d-ccg4v"] Sep 15 21:35:34.365907 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.365872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshz5\" (UniqueName: \"kubernetes.io/projected/41470267-6629-4638-9791-670ab181a13a-kube-api-access-gshz5\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.366076 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.365931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-metrics-certs\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.366076 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.366008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-webhook-cert\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.389796 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.389766 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7555b55b4d-tzchz"] Sep 15 21:35:34.393739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.393720 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.401612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.401592 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-tzchz"] Sep 15 21:35:34.467327 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.467293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshz5\" (UniqueName: \"kubernetes.io/projected/41470267-6629-4638-9791-670ab181a13a-kube-api-access-gshz5\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.467327 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.467336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-metrics-certs\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.467566 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.467388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-webhook-cert\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.469900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.469878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-metrics-certs\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.470061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.470043 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-webhook-cert\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.476154 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.476131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshz5\" (UniqueName: \"kubernetes.io/projected/41470267-6629-4638-9791-670ab181a13a-kube-api-access-gshz5\") pod \"maas-controller-98cf4f94d-ccg4v\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.555236 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.555151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:34.568502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.568469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-webhook-cert\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.568638 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.568531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-metrics-certs\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.568705 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.568686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sxs\" (UniqueName: \"kubernetes.io/projected/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-kube-api-access-j4sxs\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.669795 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.669755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sxs\" (UniqueName: \"kubernetes.io/projected/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-kube-api-access-j4sxs\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.669795 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.669809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-webhook-cert\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.670039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.669946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-metrics-certs\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.672462 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.672434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-metrics-certs\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.672556 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.672434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-webhook-cert\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.678041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.678024 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sxs\" (UniqueName: \"kubernetes.io/projected/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-kube-api-access-j4sxs\") pod \"maas-controller-7555b55b4d-tzchz\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.704886 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.704863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:34.824174 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.824071 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-98cf4f94d-ccg4v"] Sep 15 21:35:34.827504 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:35:34.827477 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41470267_6629_4638_9791_670ab181a13a.slice/crio-ddf3ec8cabf305d8672a2ff806761ea8d76ed7ec9a75e603127f07f2c77d8e58 WatchSource:0}: Error finding container ddf3ec8cabf305d8672a2ff806761ea8d76ed7ec9a75e603127f07f2c77d8e58: Status 404 returned error can't find the container with id ddf3ec8cabf305d8672a2ff806761ea8d76ed7ec9a75e603127f07f2c77d8e58 Sep 15 21:35:34.972581 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:34.972516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-tzchz"] Sep 15 21:35:34.974750 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:35:34.974718 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf33e80_fe68_46f7_b9f7_14ed1e4e7627.slice/crio-60b6d253987b42d181b97df9d83ee2c7d7d3d5c1e2e251c2518e592353b82554 WatchSource:0}: Error finding container 60b6d253987b42d181b97df9d83ee2c7d7d3d5c1e2e251c2518e592353b82554: Status 404 returned error can't find the container with id 60b6d253987b42d181b97df9d83ee2c7d7d3d5c1e2e251c2518e592353b82554 Sep 15 21:35:35.542027 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:35.541951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" event={"ID":"41470267-6629-4638-9791-670ab181a13a","Type":"ContainerStarted","Data":"ddf3ec8cabf305d8672a2ff806761ea8d76ed7ec9a75e603127f07f2c77d8e58"} Sep 15 21:35:35.543901 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:35.543830 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-tzchz" event={"ID":"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627","Type":"ContainerStarted","Data":"60b6d253987b42d181b97df9d83ee2c7d7d3d5c1e2e251c2518e592353b82554"} Sep 15 21:35:37.555056 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:37.555023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" event={"ID":"41470267-6629-4638-9791-670ab181a13a","Type":"ContainerStarted","Data":"738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce"} Sep 15 21:35:37.555512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:37.555093 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:37.556541 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:37.556520 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-tzchz" event={"ID":"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627","Type":"ContainerStarted","Data":"d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548"} Sep 15 21:35:37.556755 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:37.556564 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:37.589356 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:37.589146 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" podStartSLOduration=0.981044488 podStartE2EDuration="3.589123948s" totalImagesPullingTime="2.60807946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:37.570857684 +0000 UTC m=+586.910337135" watchObservedRunningTime="2026-09-15 21:35:37.589123948 +0000 UTC m=+586.928603401" Sep 15 21:35:37.589659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:37.589302 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7555b55b4d-tzchz" podStartSLOduration=1.118318203 podStartE2EDuration="3.589295152s" totalImagesPullingTime="2.470976949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:37.585523898 +0000 UTC m=+586.925003349" watchObservedRunningTime="2026-09-15 21:35:37.589295152 +0000 UTC m=+586.928774605" Sep 15 21:35:48.562893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.562859 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 21:35:48.563454 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.563225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:48.638296 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.638265 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-98cf4f94d-ccg4v"] Sep 15 21:35:48.638479 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.638459 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" podUID="41470267-6629-4638-9791-670ab181a13a" containerName="manager" containerID="cri-o://738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce" gracePeriod=30 Sep 15 21:35:48.891131 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.891081 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:48.985117 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.985074 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gshz5\" (UniqueName: \"kubernetes.io/projected/41470267-6629-4638-9791-670ab181a13a-kube-api-access-gshz5\") pod \"41470267-6629-4638-9791-670ab181a13a\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " Sep 15 21:35:48.985269 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.985159 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-webhook-cert\") pod \"41470267-6629-4638-9791-670ab181a13a\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " Sep 15 21:35:48.985269 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.985194 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-metrics-certs\") pod \"41470267-6629-4638-9791-670ab181a13a\" (UID: \"41470267-6629-4638-9791-670ab181a13a\") " Sep 15 21:35:48.987422 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.987392 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "41470267-6629-4638-9791-670ab181a13a" (UID: "41470267-6629-4638-9791-670ab181a13a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:35:48.987517 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.987471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41470267-6629-4638-9791-670ab181a13a-kube-api-access-gshz5" (OuterVolumeSpecName: "kube-api-access-gshz5") pod "41470267-6629-4638-9791-670ab181a13a" (UID: "41470267-6629-4638-9791-670ab181a13a"). InnerVolumeSpecName "kube-api-access-gshz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:35:48.987517 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:48.987495 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41470267-6629-4638-9791-670ab181a13a" (UID: "41470267-6629-4638-9791-670ab181a13a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:35:49.085938 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.085904 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshz5\" (UniqueName: \"kubernetes.io/projected/41470267-6629-4638-9791-670ab181a13a-kube-api-access-gshz5\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:49.085938 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.085932 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-webhook-cert\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:49.085938 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.085943 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41470267-6629-4638-9791-670ab181a13a-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:35:49.612853 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.612821 2575 generic.go:356] "Generic (PLEG): container finished" podID="41470267-6629-4638-9791-670ab181a13a" containerID="738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce" exitCode=0 Sep 15 21:35:49.613289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.612871 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" Sep 15 21:35:49.613289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.612902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" event={"ID":"41470267-6629-4638-9791-670ab181a13a","Type":"ContainerDied","Data":"738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce"} Sep 15 21:35:49.613289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.612959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98cf4f94d-ccg4v" event={"ID":"41470267-6629-4638-9791-670ab181a13a","Type":"ContainerDied","Data":"ddf3ec8cabf305d8672a2ff806761ea8d76ed7ec9a75e603127f07f2c77d8e58"} Sep 15 21:35:49.613289 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.612989 2575 scope.go:117] "RemoveContainer" containerID="738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce" Sep 15 21:35:49.621335 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.621312 2575 scope.go:117] "RemoveContainer" containerID="738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce" Sep 15 21:35:49.621610 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:35:49.621593 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce\": container with ID starting with 738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce not found: ID does not exist" containerID="738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce" Sep 15 21:35:49.621672 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.621621 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce"} err="failed to get container status \"738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce\": rpc error: code = NotFound desc = could not find container \"738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce\": container with ID starting with 738c7c9a459924249919660bc667db69be374ea856c2d06b2cb28524e516b9ce not found: ID does not exist" Sep 15 21:35:49.629360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.629337 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-98cf4f94d-ccg4v"] Sep 15 21:35:49.632533 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:49.632509 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-98cf4f94d-ccg4v"] Sep 15 21:35:51.147233 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:35:51.147195 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41470267-6629-4638-9791-670ab181a13a" path="/var/lib/kubelet/pods/41470267-6629-4638-9791-670ab181a13a/volumes" Sep 15 21:36:10.790019 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.789985 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs"] Sep 15 21:36:10.790559 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.790496 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41470267-6629-4638-9791-670ab181a13a" containerName="manager" Sep 15 21:36:10.790559 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.790513 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="41470267-6629-4638-9791-670ab181a13a" containerName="manager" Sep 15 21:36:10.790633 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.790600 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="41470267-6629-4638-9791-670ab181a13a" containerName="manager" Sep 15 21:36:10.807762 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.807723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs"] Sep 15 21:36:10.807919 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.807834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.810304 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.810280 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 15 21:36:10.811072 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.811054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bmlq\"" Sep 15 21:36:10.811204 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.811067 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 21:36:10.874582 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.874553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-openshift-service-ca\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.874582 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.874583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-metrics-certs\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.874757 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.874605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.874757 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.874680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpxv\" (UniqueName: \"kubernetes.io/projected/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-kube-api-access-llpxv\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.975687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.975652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-openshift-service-ca\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.975840 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.975694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-metrics-certs\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.975840 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.975720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.975840 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.975781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpxv\" (UniqueName: \"kubernetes.io/projected/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-kube-api-access-llpxv\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.976014 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:36:10.975891 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 15 21:36:10.976014 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:36:10.975976 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls podName:c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05 nodeName:}" failed. No retries permitted until 2026-09-15 21:36:11.475955473 +0000 UTC m=+620.815434910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls") pod "maas-api-5557cbddd4-cvfxs" (UID: "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05") : secret "maas-api-serving-cert" not found Sep 15 21:36:10.976436 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.976417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-openshift-service-ca\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.978444 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.978426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-metrics-certs\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:10.984996 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:10.984978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpxv\" (UniqueName: \"kubernetes.io/projected/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-kube-api-access-llpxv\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:11.479922 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:11.479885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:11.484859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:11.484829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls\") pod \"maas-api-5557cbddd4-cvfxs\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:11.720996 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:11.720960 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:11.997375 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:11.997345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs"] Sep 15 21:36:12.000072 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:36:12.000034 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a42ef4_a4c6_4a38_8667_d1c5b4ce9e05.slice/crio-a4a6593dc6da1088324280453699210f052eed9669ca9154ac6fc53635ef5b13 WatchSource:0}: Error finding container a4a6593dc6da1088324280453699210f052eed9669ca9154ac6fc53635ef5b13: Status 404 returned error can't find the container with id a4a6593dc6da1088324280453699210f052eed9669ca9154ac6fc53635ef5b13 Sep 15 21:36:12.724343 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:12.724313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" event={"ID":"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05","Type":"ContainerStarted","Data":"a4a6593dc6da1088324280453699210f052eed9669ca9154ac6fc53635ef5b13"} Sep 15 21:36:13.732053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:13.731960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" event={"ID":"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05","Type":"ContainerStarted","Data":"cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56"} Sep 15 21:36:13.732053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:13.732037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:13.748435 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:13.748390 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" podStartSLOduration=2.286687913 podStartE2EDuration="3.748375676s" totalImagesPullingTime="1.461687763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:13.747343163 +0000 UTC m=+623.086822628" watchObservedRunningTime="2026-09-15 21:36:13.748375676 +0000 UTC m=+623.087855127" Sep 15 21:36:19.742781 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:19.742753 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 21:36:51.632518 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.632477 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47"] Sep 15 21:36:51.634897 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.634878 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.637739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.637712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 21:36:51.637893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.637773 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 21:36:51.638937 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.638916 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r5v6f\"" Sep 15 21:36:51.639042 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.638984 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 15 21:36:51.644611 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.644584 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47"] Sep 15 21:36:51.720194 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.720149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.720194 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.720198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02954641-80ba-4628-99a9-bade4aa2f9a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.720434 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.720276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.720434 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.720311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbbg\" (UniqueName: \"kubernetes.io/projected/02954641-80ba-4628-99a9-bade4aa2f9a8-kube-api-access-kfbbg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.720434 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.720355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.720434 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.720370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821311 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821497 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbbg\" (UniqueName: \"kubernetes.io/projected/02954641-80ba-4628-99a9-bade4aa2f9a8-kube-api-access-kfbbg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821497 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821497 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821497 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821750 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02954641-80ba-4628-99a9-bade4aa2f9a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821750 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.821920 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.822022 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.821998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.824082 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.824054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02954641-80ba-4628-99a9-bade4aa2f9a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.824082 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.824067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02954641-80ba-4628-99a9-bade4aa2f9a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.828627 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.828602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbbg\" (UniqueName: \"kubernetes.io/projected/02954641-80ba-4628-99a9-bade4aa2f9a8-kube-api-access-kfbbg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vwn47\" (UID: \"02954641-80ba-4628-99a9-bade4aa2f9a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:51.946477 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:51.946433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:36:52.219645 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:52.219561 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47"] Sep 15 21:36:52.222477 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:36:52.222449 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02954641_80ba_4628_99a9_bade4aa2f9a8.slice/crio-4bdbdceada6ab123704494a189fc40af69ab6a8436a1739ba96d98e686ad5837 WatchSource:0}: Error finding container 4bdbdceada6ab123704494a189fc40af69ab6a8436a1739ba96d98e686ad5837: Status 404 returned error can't find the container with id 4bdbdceada6ab123704494a189fc40af69ab6a8436a1739ba96d98e686ad5837 Sep 15 21:36:52.223775 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:52.223756 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:36:52.917475 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:52.917434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" event={"ID":"02954641-80ba-4628-99a9-bade4aa2f9a8","Type":"ContainerStarted","Data":"4bdbdceada6ab123704494a189fc40af69ab6a8436a1739ba96d98e686ad5837"} Sep 15 21:36:53.923038 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:53.922982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" event={"ID":"02954641-80ba-4628-99a9-bade4aa2f9a8","Type":"ContainerStarted","Data":"4a0001a17b98a571a68d4a7813556913ce765866d39577ee3f409528720901c3"} Sep 15 21:36:53.940828 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:36:53.940773 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" podStartSLOduration=1.873291509 podStartE2EDuration="2.940754667s" totalImagesPullingTime="1.067463158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:53.938927865 +0000 UTC m=+663.278407317" watchObservedRunningTime="2026-09-15 21:36:53.940754667 +0000 UTC m=+663.280234117" Sep 15 21:37:01.946971 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:01.946932 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:37:01.947389 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:01.947005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:37:01.959421 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:01.959388 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:37:01.976081 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:01.976047 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vwn47" Sep 15 21:37:16.141145 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.141090 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k"] Sep 15 21:37:16.144277 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.144248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.146660 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.146633 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 15 21:37:16.153837 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.153802 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k"] Sep 15 21:37:16.245228 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.245181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.245424 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.245253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.245502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.245438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.245502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.245469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.245593 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.245514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s586h\" (UniqueName: \"kubernetes.io/projected/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-kube-api-access-s586h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.245638 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.245603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.346993 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.346960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s586h\" (UniqueName: \"kubernetes.io/projected/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-kube-api-access-s586h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347196 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347595 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.347680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.347592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.349610 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.349559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.349719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.349710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.355073 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.355045 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s586h\" (UniqueName: \"kubernetes.io/projected/d1b88e0a-42b4-489a-8918-b6b7b32a32dd-kube-api-access-s586h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k\" (UID: \"d1b88e0a-42b4-489a-8918-b6b7b32a32dd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.455625 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.455584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:16.729745 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:16.729572 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k"] Sep 15 21:37:16.732676 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:37:16.732637 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b88e0a_42b4_489a_8918_b6b7b32a32dd.slice/crio-8e2269acfca9e27a664f0ffbe24c0dc06e17f3a948f1074f7d6e2034771b995a WatchSource:0}: Error finding container 8e2269acfca9e27a664f0ffbe24c0dc06e17f3a948f1074f7d6e2034771b995a: Status 404 returned error can't find the container with id 8e2269acfca9e27a664f0ffbe24c0dc06e17f3a948f1074f7d6e2034771b995a Sep 15 21:37:17.042386 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:17.042342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" event={"ID":"d1b88e0a-42b4-489a-8918-b6b7b32a32dd","Type":"ContainerStarted","Data":"705110c4476300bf7ed71395274b6ddbd7169175ac33d24d5a783232ffd84109"} Sep 15 21:37:17.042569 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:17.042396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" event={"ID":"d1b88e0a-42b4-489a-8918-b6b7b32a32dd","Type":"ContainerStarted","Data":"8e2269acfca9e27a664f0ffbe24c0dc06e17f3a948f1074f7d6e2034771b995a"} Sep 15 21:37:17.063323 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:17.063264 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" podStartSLOduration=0.845585969 podStartE2EDuration="1.063247737s" totalImagesPullingTime="217.661768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:17.061607917 +0000 UTC m=+686.401087367" watchObservedRunningTime="2026-09-15 21:37:17.063247737 +0000 UTC m=+686.402727189" Sep 15 21:37:26.455761 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:26.455713 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:26.456293 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:26.455791 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:26.468430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:26.468408 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:27.099644 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:27.099616 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k" Sep 15 21:37:34.531271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.531239 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5"] Sep 15 21:37:34.536032 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.536003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.539146 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.539123 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 15 21:37:34.545037 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.545010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5"] Sep 15 21:37:34.596284 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.596255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.596284 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.596290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.596510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.596313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.596510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.596331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.596510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.596408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.596510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.596461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqkl9\" (UniqueName: \"kubernetes.io/projected/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-kube-api-access-bqkl9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697088 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqkl9\" (UniqueName: \"kubernetes.io/projected/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-kube-api-access-bqkl9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697543 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697649 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.697649 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.697626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.699563 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.699543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.699850 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.699830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.705362 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.705342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqkl9\" (UniqueName: \"kubernetes.io/projected/28ac6010-8a36-4a4b-96a5-4bc8d8ee5302-kube-api-access-bqkl9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w4ng5\" (UID: \"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:34.848878 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:34.848779 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:35.128947 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:35.128919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5"] Sep 15 21:37:35.131045 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:37:35.131016 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ac6010_8a36_4a4b_96a5_4bc8d8ee5302.slice/crio-db8819f4716770ee87f4f29d75524a3cfcee35fda18151ad97ebf304f8b4349a WatchSource:0}: Error finding container db8819f4716770ee87f4f29d75524a3cfcee35fda18151ad97ebf304f8b4349a: Status 404 returned error can't find the container with id db8819f4716770ee87f4f29d75524a3cfcee35fda18151ad97ebf304f8b4349a Sep 15 21:37:36.131862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:36.131831 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" event={"ID":"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302","Type":"ContainerStarted","Data":"9ecbdb1a5892180e1c71c859e14790d5fea7b381662d8ee61f7dba818647abd6"} Sep 15 21:37:36.132278 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:36.131879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" event={"ID":"28ac6010-8a36-4a4b-96a5-4bc8d8ee5302","Type":"ContainerStarted","Data":"db8819f4716770ee87f4f29d75524a3cfcee35fda18151ad97ebf304f8b4349a"} Sep 15 21:37:36.151597 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:36.151548 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" podStartSLOduration=1.96740687 podStartE2EDuration="2.151533792s" totalImagesPullingTime="184.126922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:36.149821141 +0000 UTC m=+705.489300593" watchObservedRunningTime="2026-09-15 21:37:36.151533792 +0000 UTC m=+705.491013266" Sep 15 21:37:44.849869 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:44.849780 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:44.849869 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:44.849843 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:44.862447 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:44.862416 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:37:45.187265 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:37:45.187240 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w4ng5" Sep 15 21:39:10.972721 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:10.972683 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-558d748df4-5rq7c"] Sep 15 21:39:10.976081 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:10.976053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:10.984278 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:10.984250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558d748df4-5rq7c"] Sep 15 21:39:11.022987 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.022957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aafbc96c-896a-4a85-aed1-2b56f551a38d-tls-cert\") pod \"authorino-558d748df4-5rq7c\" (UID: \"aafbc96c-896a-4a85-aed1-2b56f551a38d\") " pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.022987 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.022993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrpzp\" (UniqueName: \"kubernetes.io/projected/aafbc96c-896a-4a85-aed1-2b56f551a38d-kube-api-access-mrpzp\") pod \"authorino-558d748df4-5rq7c\" (UID: \"aafbc96c-896a-4a85-aed1-2b56f551a38d\") " pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.123739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.123695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aafbc96c-896a-4a85-aed1-2b56f551a38d-tls-cert\") pod \"authorino-558d748df4-5rq7c\" (UID: \"aafbc96c-896a-4a85-aed1-2b56f551a38d\") " pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.123921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.123747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpzp\" (UniqueName: \"kubernetes.io/projected/aafbc96c-896a-4a85-aed1-2b56f551a38d-kube-api-access-mrpzp\") pod \"authorino-558d748df4-5rq7c\" (UID: \"aafbc96c-896a-4a85-aed1-2b56f551a38d\") " pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.126463 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.126433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aafbc96c-896a-4a85-aed1-2b56f551a38d-tls-cert\") pod \"authorino-558d748df4-5rq7c\" (UID: \"aafbc96c-896a-4a85-aed1-2b56f551a38d\") " pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.132838 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.132805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpzp\" (UniqueName: \"kubernetes.io/projected/aafbc96c-896a-4a85-aed1-2b56f551a38d-kube-api-access-mrpzp\") pod \"authorino-558d748df4-5rq7c\" (UID: \"aafbc96c-896a-4a85-aed1-2b56f551a38d\") " pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.286381 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.286301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558d748df4-5rq7c" Sep 15 21:39:11.558364 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.558339 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558d748df4-5rq7c"] Sep 15 21:39:11.560337 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:11.560312 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaafbc96c_896a_4a85_aed1_2b56f551a38d.slice/crio-2d85e108a0bac06fc90a8aea0f5eddf494b55591afd3fa889f404e004820f431 WatchSource:0}: Error finding container 2d85e108a0bac06fc90a8aea0f5eddf494b55591afd3fa889f404e004820f431: Status 404 returned error can't find the container with id 2d85e108a0bac06fc90a8aea0f5eddf494b55591afd3fa889f404e004820f431 Sep 15 21:39:11.568569 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:11.568540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558d748df4-5rq7c" event={"ID":"aafbc96c-896a-4a85-aed1-2b56f551a38d","Type":"ContainerStarted","Data":"2d85e108a0bac06fc90a8aea0f5eddf494b55591afd3fa889f404e004820f431"} Sep 15 21:39:12.576956 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.576858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558d748df4-5rq7c" event={"ID":"aafbc96c-896a-4a85-aed1-2b56f551a38d","Type":"ContainerStarted","Data":"54f0c03c18bc3af4dc07234b52c5222849d6c1893451bb8ccb343a8641b5dc9e"} Sep 15 21:39:12.594638 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.594573 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-558d748df4-5rq7c" podStartSLOduration=1.8508904689999999 podStartE2EDuration="2.594552074s" totalImagesPullingTime="743.661605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:12.591192933 +0000 UTC m=+801.930672387" watchObservedRunningTime="2026-09-15 21:39:12.594552074 +0000 UTC m=+801.934031527" Sep 15 21:39:12.625863 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.624776 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dbcb84b8c-zrv8m"] Sep 15 21:39:12.625863 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.624986 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" podUID="ed89d270-f7e8-470d-aba3-44772b708246" containerName="authorino" containerID="cri-o://cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e" gracePeriod=30 Sep 15 21:39:12.878200 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.878175 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:39:12.939996 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.939965 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed89d270-f7e8-470d-aba3-44772b708246-tls-cert\") pod \"ed89d270-f7e8-470d-aba3-44772b708246\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " Sep 15 21:39:12.940180 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.940059 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj6s7\" (UniqueName: \"kubernetes.io/projected/ed89d270-f7e8-470d-aba3-44772b708246-kube-api-access-gj6s7\") pod \"ed89d270-f7e8-470d-aba3-44772b708246\" (UID: \"ed89d270-f7e8-470d-aba3-44772b708246\") " Sep 15 21:39:12.942260 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.942230 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed89d270-f7e8-470d-aba3-44772b708246-kube-api-access-gj6s7" (OuterVolumeSpecName: "kube-api-access-gj6s7") pod "ed89d270-f7e8-470d-aba3-44772b708246" (UID: "ed89d270-f7e8-470d-aba3-44772b708246"). InnerVolumeSpecName "kube-api-access-gj6s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:39:12.951806 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:12.951774 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed89d270-f7e8-470d-aba3-44772b708246-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ed89d270-f7e8-470d-aba3-44772b708246" (UID: "ed89d270-f7e8-470d-aba3-44772b708246"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:39:13.041539 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.041510 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed89d270-f7e8-470d-aba3-44772b708246-tls-cert\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:39:13.041539 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.041536 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6s7\" (UniqueName: \"kubernetes.io/projected/ed89d270-f7e8-470d-aba3-44772b708246-kube-api-access-gj6s7\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:39:13.585317 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.585280 2575 generic.go:356] "Generic (PLEG): container finished" podID="ed89d270-f7e8-470d-aba3-44772b708246" containerID="cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e" exitCode=0 Sep 15 21:39:13.585759 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.585328 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" Sep 15 21:39:13.585759 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.585370 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" event={"ID":"ed89d270-f7e8-470d-aba3-44772b708246","Type":"ContainerDied","Data":"cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e"} Sep 15 21:39:13.585759 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.585418 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbcb84b8c-zrv8m" event={"ID":"ed89d270-f7e8-470d-aba3-44772b708246","Type":"ContainerDied","Data":"a893e08db56bc9a4aca9d9adb24208b464f4d277f00854f433eaded303c99cea"} Sep 15 21:39:13.585759 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.585442 2575 scope.go:117] "RemoveContainer" containerID="cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e" Sep 15 21:39:13.594634 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.594609 2575 scope.go:117] "RemoveContainer" containerID="cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e" Sep 15 21:39:13.594906 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:39:13.594880 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e\": container with ID starting with cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e not found: ID does not exist" containerID="cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e" Sep 15 21:39:13.594971 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.594905 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e"} err="failed to get container status \"cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e\": rpc error: code = NotFound desc = could not find container \"cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e\": container with ID starting with cb5a858953c91d561ef7fa7a291dbe025a17a391239738fcef674dd46b806e0e not found: ID does not exist" Sep 15 21:39:13.599758 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.599735 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dbcb84b8c-zrv8m"] Sep 15 21:39:13.603518 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:13.603494 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dbcb84b8c-zrv8m"] Sep 15 21:39:15.146838 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:15.146803 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed89d270-f7e8-470d-aba3-44772b708246" path="/var/lib/kubelet/pods/ed89d270-f7e8-470d-aba3-44772b708246/volumes" Sep 15 21:39:35.548835 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.548799 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5"] Sep 15 21:39:35.549316 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.549189 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed89d270-f7e8-470d-aba3-44772b708246" containerName="authorino" Sep 15 21:39:35.549316 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.549202 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed89d270-f7e8-470d-aba3-44772b708246" containerName="authorino" Sep 15 21:39:35.549434 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.549341 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed89d270-f7e8-470d-aba3-44772b708246" containerName="authorino" Sep 15 21:39:35.552895 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.552870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.555497 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.555455 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1af1db-serving-cert\"" Sep 15 21:39:35.563618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.563074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5"] Sep 15 21:39:35.646647 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.646611 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.646855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.646691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcn7t\" (UniqueName: \"kubernetes.io/projected/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-kube-api-access-mcn7t\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.646855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.646758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.646855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.646805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-metrics-certs\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.748254 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.748216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcn7t\" (UniqueName: \"kubernetes.io/projected/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-kube-api-access-mcn7t\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.748484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.748281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.748484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.748327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-metrics-certs\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.748484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.748352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.748484 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:39:35.748455 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-serving-cert: secret "maas-api-e2e-inf-a-1af1db-serving-cert" not found Sep 15 21:39:35.748692 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:39:35.748530 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls podName:a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec nodeName:}" failed. No retries permitted until 2026-09-15 21:39:36.24851166 +0000 UTC m=+825.587991108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls") pod "maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" (UID: "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec") : secret "maas-api-e2e-inf-a-1af1db-serving-cert" not found Sep 15 21:39:35.749054 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.749031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.750967 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.750948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-metrics-certs\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:35.756611 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:35.756590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcn7t\" (UniqueName: \"kubernetes.io/projected/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-kube-api-access-mcn7t\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:36.252822 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.252789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:36.255299 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.255280 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:36.467433 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.467406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:36.545458 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.545429 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv"] Sep 15 21:39:36.549449 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.549425 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.551801 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.551665 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-360e43-serving-cert\"" Sep 15 21:39:36.560357 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.560329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv"] Sep 15 21:39:36.657072 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.657033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b991f35-71f9-4422-9896-068e62662be0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.657274 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.657088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-metrics-certs\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.657274 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.657190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwr9v\" (UniqueName: \"kubernetes.io/projected/8b991f35-71f9-4422-9896-068e62662be0-kube-api-access-cwr9v\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.657274 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.657223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.758264 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.757889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwr9v\" (UniqueName: \"kubernetes.io/projected/8b991f35-71f9-4422-9896-068e62662be0-kube-api-access-cwr9v\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.758264 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.757963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.758264 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.758025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b991f35-71f9-4422-9896-068e62662be0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.758264 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.758065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-metrics-certs\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.758694 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:39:36.758479 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-serving-cert: secret "maas-api-e2e-worker-w0-360e43-serving-cert" not found Sep 15 21:39:36.758957 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.758897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b991f35-71f9-4422-9896-068e62662be0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.759167 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:39:36.759153 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls podName:8b991f35-71f9-4422-9896-068e62662be0 nodeName:}" failed. No retries permitted until 2026-09-15 21:39:37.259129424 +0000 UTC m=+826.598608859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls") pod "maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" (UID: "8b991f35-71f9-4422-9896-068e62662be0") : secret "maas-api-e2e-worker-w0-360e43-serving-cert" not found Sep 15 21:39:36.760871 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.760842 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5"] Sep 15 21:39:36.761290 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.761274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-metrics-certs\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:36.764542 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:36.764520 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f11aa0_2a2f_4ab7_a41e_4f52fddab9ec.slice/crio-123b44d5fc3623bee722656a957c20496ddeb364842ed1a94880562596ef3cb4 WatchSource:0}: Error finding container 123b44d5fc3623bee722656a957c20496ddeb364842ed1a94880562596ef3cb4: Status 404 returned error can't find the container with id 123b44d5fc3623bee722656a957c20496ddeb364842ed1a94880562596ef3cb4 Sep 15 21:39:36.784533 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:36.784473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwr9v\" (UniqueName: \"kubernetes.io/projected/8b991f35-71f9-4422-9896-068e62662be0-kube-api-access-cwr9v\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:37.261739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.261706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:37.264712 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.264686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:37.462546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.462504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:37.632476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.632439 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx"] Sep 15 21:39:37.637771 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.637749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.640116 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.640077 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-56e305-serving-cert\"" Sep 15 21:39:37.645300 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.645261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx"] Sep 15 21:39:37.705022 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.704987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" event={"ID":"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec","Type":"ContainerStarted","Data":"67bb6f928906bf412e48195418ea87b0dbff1b144b35e8ce6661f15c8d617453"} Sep 15 21:39:37.705226 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.705044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" event={"ID":"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec","Type":"ContainerStarted","Data":"123b44d5fc3623bee722656a957c20496ddeb364842ed1a94880562596ef3cb4"} Sep 15 21:39:37.705226 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.705131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:37.722763 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.722707 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" podStartSLOduration=2.258195887 podStartE2EDuration="2.722688794s" totalImagesPullingTime="464.492907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:37.720837547 +0000 UTC m=+827.060316999" watchObservedRunningTime="2026-09-15 21:39:37.722688794 +0000 UTC m=+827.062168245" Sep 15 21:39:37.743431 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.743407 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv"] Sep 15 21:39:37.745873 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:37.745846 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b991f35_71f9_4422_9896_068e62662be0.slice/crio-25e5753ce37d83bae45160ea8a28dc6ed23637737d961c1dce05ede8b7481c19 WatchSource:0}: Error finding container 25e5753ce37d83bae45160ea8a28dc6ed23637737d961c1dce05ede8b7481c19: Status 404 returned error can't find the container with id 25e5753ce37d83bae45160ea8a28dc6ed23637737d961c1dce05ede8b7481c19 Sep 15 21:39:37.767149 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.767046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pc8\" (UniqueName: \"kubernetes.io/projected/72789745-49ef-426a-aa3e-f806c8420b99-kube-api-access-44pc8\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.767149 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.767123 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72789745-49ef-426a-aa3e-f806c8420b99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.767328 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.767157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.767328 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.767201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-metrics-certs\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.867836 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.867796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-metrics-certs\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.868009 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.867890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pc8\" (UniqueName: \"kubernetes.io/projected/72789745-49ef-426a-aa3e-f806c8420b99-kube-api-access-44pc8\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.868009 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.867920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72789745-49ef-426a-aa3e-f806c8420b99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.868009 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.867938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.868667 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.868579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72789745-49ef-426a-aa3e-f806c8420b99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.870477 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.870449 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.870574 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.870510 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-metrics-certs\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.876661 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.876635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pc8\" (UniqueName: \"kubernetes.io/projected/72789745-49ef-426a-aa3e-f806c8420b99-kube-api-access-44pc8\") pod \"maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:37.951814 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:37.951776 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:38.235605 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:38.235571 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx"] Sep 15 21:39:38.237363 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:38.237333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72789745_49ef_426a_aa3e_f806c8420b99.slice/crio-79bc3e7ae047c28f9c61a0a24612dc452ad13f9150916560e4bc6f62e441b276 WatchSource:0}: Error finding container 79bc3e7ae047c28f9c61a0a24612dc452ad13f9150916560e4bc6f62e441b276: Status 404 returned error can't find the container with id 79bc3e7ae047c28f9c61a0a24612dc452ad13f9150916560e4bc6f62e441b276 Sep 15 21:39:38.713332 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:38.713296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" event={"ID":"8b991f35-71f9-4422-9896-068e62662be0","Type":"ContainerStarted","Data":"71d1b61cb5167dda69c040d975d0df05b6ddd5d0bd468798d0b7927627ba69a5"} Sep 15 21:39:38.713718 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:38.713505 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:38.713718 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:38.713550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" event={"ID":"8b991f35-71f9-4422-9896-068e62662be0","Type":"ContainerStarted","Data":"25e5753ce37d83bae45160ea8a28dc6ed23637737d961c1dce05ede8b7481c19"} Sep 15 21:39:38.714802 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:38.714776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" event={"ID":"72789745-49ef-426a-aa3e-f806c8420b99","Type":"ContainerStarted","Data":"79bc3e7ae047c28f9c61a0a24612dc452ad13f9150916560e4bc6f62e441b276"} Sep 15 21:39:38.734919 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:38.734869 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" podStartSLOduration=2.173614801 podStartE2EDuration="2.734853939s" totalImagesPullingTime="561.239138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:38.732760351 +0000 UTC m=+828.072239803" watchObservedRunningTime="2026-09-15 21:39:38.734853939 +0000 UTC m=+828.074333389" Sep 15 21:39:39.721087 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:39.721047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" event={"ID":"72789745-49ef-426a-aa3e-f806c8420b99","Type":"ContainerStarted","Data":"5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697"} Sep 15 21:39:39.721506 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:39.721167 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:39.738068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:39.738008 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" podStartSLOduration=2.2977064719999998 podStartE2EDuration="2.737987968s" totalImagesPullingTime="440.281496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:39.737888876 +0000 UTC m=+829.077368332" watchObservedRunningTime="2026-09-15 21:39:39.737987968 +0000 UTC m=+829.077467445" Sep 15 21:39:43.720546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:43.720515 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:39:44.726775 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:44.726747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:39:45.732518 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:45.732487 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:39:45.951356 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:45.951322 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz"] Sep 15 21:39:45.955173 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:45.955151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:45.957440 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:45.957423 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-5416b7-serving-cert\"" Sep 15 21:39:45.966075 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:45.966050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz"] Sep 15 21:39:46.050778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.050695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vn9b\" (UniqueName: \"kubernetes.io/projected/86a7d18b-19aa-47da-915e-d8c03b812a0f-kube-api-access-2vn9b\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.050921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.050791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-metrics-certs\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.050921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.050866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-maas-api-tls\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.050921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.050884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7d18b-19aa-47da-915e-d8c03b812a0f-openshift-service-ca\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.151660 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.151623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-metrics-certs\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.151851 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.151705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-maas-api-tls\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.151851 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.151733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7d18b-19aa-47da-915e-d8c03b812a0f-openshift-service-ca\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.151851 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.151786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vn9b\" (UniqueName: \"kubernetes.io/projected/86a7d18b-19aa-47da-915e-d8c03b812a0f-kube-api-access-2vn9b\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.152447 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.152425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7d18b-19aa-47da-915e-d8c03b812a0f-openshift-service-ca\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.154221 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.154202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-metrics-certs\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.154308 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.154253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-maas-api-tls\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.160542 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.160522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vn9b\" (UniqueName: \"kubernetes.io/projected/86a7d18b-19aa-47da-915e-d8c03b812a0f-kube-api-access-2vn9b\") pod \"maas-api-e2e-autores-5416b7-cb55f4c95-9npsz\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.268546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.268513 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:46.544498 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.544469 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz"] Sep 15 21:39:46.546814 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:46.546781 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a7d18b_19aa_47da_915e_d8c03b812a0f.slice/crio-16d550dabadb3c8975533a09a81f94e81936a57dcfe6f431470586138d5015ec WatchSource:0}: Error finding container 16d550dabadb3c8975533a09a81f94e81936a57dcfe6f431470586138d5015ec: Status 404 returned error can't find the container with id 16d550dabadb3c8975533a09a81f94e81936a57dcfe6f431470586138d5015ec Sep 15 21:39:46.756437 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.756406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" event={"ID":"86a7d18b-19aa-47da-915e-d8c03b812a0f","Type":"ContainerStarted","Data":"16d550dabadb3c8975533a09a81f94e81936a57dcfe6f431470586138d5015ec"} Sep 15 21:39:46.926583 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.926552 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv"] Sep 15 21:39:46.931772 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.931748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:46.934200 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.934135 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ba8d384-serving-cert\"" Sep 15 21:39:46.946681 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:46.946655 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv"] Sep 15 21:39:47.060377 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.060297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrgj\" (UniqueName: \"kubernetes.io/projected/0de79225-c20d-4bba-991d-721309c70c42-kube-api-access-xjrgj\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.060377 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.060345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de79225-c20d-4bba-991d-721309c70c42-openshift-service-ca\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.060551 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.060406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-metrics-certs\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.060551 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.060485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-maas-api-tls\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.161639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.161604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de79225-c20d-4bba-991d-721309c70c42-openshift-service-ca\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.161639 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.161637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-metrics-certs\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.161818 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.161687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-maas-api-tls\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.161818 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.161765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrgj\" (UniqueName: \"kubernetes.io/projected/0de79225-c20d-4bba-991d-721309c70c42-kube-api-access-xjrgj\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.162318 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.162293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de79225-c20d-4bba-991d-721309c70c42-openshift-service-ca\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.164304 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.164280 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-metrics-certs\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.164428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.164331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-maas-api-tls\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.172924 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.172904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrgj\" (UniqueName: \"kubernetes.io/projected/0de79225-c20d-4bba-991d-721309c70c42-kube-api-access-xjrgj\") pod \"maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.248803 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.248766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:47.524468 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.524446 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv"] Sep 15 21:39:47.526550 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:47.526515 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de79225_c20d_4bba_991d_721309c70c42.slice/crio-91244f98763bbecfd877367ec8cbae5799286c579839c878eb40572f9ecf681d WatchSource:0}: Error finding container 91244f98763bbecfd877367ec8cbae5799286c579839c878eb40572f9ecf681d: Status 404 returned error can't find the container with id 91244f98763bbecfd877367ec8cbae5799286c579839c878eb40572f9ecf681d Sep 15 21:39:47.763656 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.763607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" event={"ID":"86a7d18b-19aa-47da-915e-d8c03b812a0f","Type":"ContainerStarted","Data":"e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb"} Sep 15 21:39:47.764125 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.763811 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:47.765067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.765043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" event={"ID":"0de79225-c20d-4bba-991d-721309c70c42","Type":"ContainerStarted","Data":"91244f98763bbecfd877367ec8cbae5799286c579839c878eb40572f9ecf681d"} Sep 15 21:39:47.779267 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:47.779158 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" podStartSLOduration=2.240275257 podStartE2EDuration="2.779137994s" totalImagesPullingTime="538.862737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:47.778627315 +0000 UTC m=+837.118106766" watchObservedRunningTime="2026-09-15 21:39:47.779137994 +0000 UTC m=+837.118617442" Sep 15 21:39:48.771589 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:48.771552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" event={"ID":"0de79225-c20d-4bba-991d-721309c70c42","Type":"ContainerStarted","Data":"340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995"} Sep 15 21:39:48.791053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:48.790996 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" podStartSLOduration=2.203755366 podStartE2EDuration="2.790978685s" totalImagesPullingTime="587.223319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:48.789068583 +0000 UTC m=+838.128548037" watchObservedRunningTime="2026-09-15 21:39:48.790978685 +0000 UTC m=+838.130458157" Sep 15 21:39:49.776182 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:49.776151 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:53.778247 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:53.778214 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:39:55.127343 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.127253 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w"] Sep 15 21:39:55.133006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.132979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.135338 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.135313 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a2d9c4-serving-cert\"" Sep 15 21:39:55.154751 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.154710 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w"] Sep 15 21:39:55.238053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.238019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlph\" (UniqueName: \"kubernetes.io/projected/b0ae0d9c-12a4-4822-a256-78d731730512-kube-api-access-vwlph\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.238251 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.238146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.238251 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.238199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0d9c-12a4-4822-a256-78d731730512-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.238251 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.238228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-metrics-certs\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.339687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.339646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.339880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.339717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0d9c-12a4-4822-a256-78d731730512-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.339880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.339751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-metrics-certs\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.339880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.339817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlph\" (UniqueName: \"kubernetes.io/projected/b0ae0d9c-12a4-4822-a256-78d731730512-kube-api-access-vwlph\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.340488 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.340461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0d9c-12a4-4822-a256-78d731730512-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.342416 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.342387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.342510 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.342416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-metrics-certs\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.347252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.347232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlph\" (UniqueName: \"kubernetes.io/projected/b0ae0d9c-12a4-4822-a256-78d731730512-kube-api-access-vwlph\") pod \"maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.451790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.451748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:55.730882 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.730804 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w"] Sep 15 21:39:55.734744 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:39:55.734713 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ae0d9c_12a4_4822_a256_78d731730512.slice/crio-aaf68a82446c052c5da0f8dabf0a3f6fbac3e97a031c5d07a514f81c79f026a6 WatchSource:0}: Error finding container aaf68a82446c052c5da0f8dabf0a3f6fbac3e97a031c5d07a514f81c79f026a6: Status 404 returned error can't find the container with id aaf68a82446c052c5da0f8dabf0a3f6fbac3e97a031c5d07a514f81c79f026a6 Sep 15 21:39:55.788443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.788415 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:39:55.810871 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:55.810835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" event={"ID":"b0ae0d9c-12a4-4822-a256-78d731730512","Type":"ContainerStarted","Data":"aaf68a82446c052c5da0f8dabf0a3f6fbac3e97a031c5d07a514f81c79f026a6"} Sep 15 21:39:56.817792 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:56.817688 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" event={"ID":"b0ae0d9c-12a4-4822-a256-78d731730512","Type":"ContainerStarted","Data":"3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4"} Sep 15 21:39:56.818368 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:56.817912 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:39:56.837415 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:39:56.837368 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" podStartSLOduration=1.414521807 podStartE2EDuration="1.837353112s" totalImagesPullingTime="422.831305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:56.836410489 +0000 UTC m=+846.175889941" watchObservedRunningTime="2026-09-15 21:39:56.837353112 +0000 UTC m=+846.176832563" Sep 15 21:40:02.829209 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:02.829173 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:40:11.012496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.012460 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv"] Sep 15 21:40:11.012950 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.012681 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" podUID="0de79225-c20d-4bba-991d-721309c70c42" containerName="maas-api" containerID="cri-o://340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995" gracePeriod=30 Sep 15 21:40:11.270641 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.270575 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:40:11.383010 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.382976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjrgj\" (UniqueName: \"kubernetes.io/projected/0de79225-c20d-4bba-991d-721309c70c42-kube-api-access-xjrgj\") pod \"0de79225-c20d-4bba-991d-721309c70c42\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " Sep 15 21:40:11.383219 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.383091 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de79225-c20d-4bba-991d-721309c70c42-openshift-service-ca\") pod \"0de79225-c20d-4bba-991d-721309c70c42\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " Sep 15 21:40:11.383219 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.383210 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-metrics-certs\") pod \"0de79225-c20d-4bba-991d-721309c70c42\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " Sep 15 21:40:11.383377 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.383243 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-maas-api-tls\") pod \"0de79225-c20d-4bba-991d-721309c70c42\" (UID: \"0de79225-c20d-4bba-991d-721309c70c42\") " Sep 15 21:40:11.383900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.383464 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de79225-c20d-4bba-991d-721309c70c42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0de79225-c20d-4bba-991d-721309c70c42" (UID: "0de79225-c20d-4bba-991d-721309c70c42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:40:11.383900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.383869 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de79225-c20d-4bba-991d-721309c70c42-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:11.386068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.386048 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0de79225-c20d-4bba-991d-721309c70c42" (UID: "0de79225-c20d-4bba-991d-721309c70c42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:40:11.386200 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.386068 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de79225-c20d-4bba-991d-721309c70c42-kube-api-access-xjrgj" (OuterVolumeSpecName: "kube-api-access-xjrgj") pod "0de79225-c20d-4bba-991d-721309c70c42" (UID: "0de79225-c20d-4bba-991d-721309c70c42"). InnerVolumeSpecName "kube-api-access-xjrgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:40:11.386200 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.386116 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0de79225-c20d-4bba-991d-721309c70c42" (UID: "0de79225-c20d-4bba-991d-721309c70c42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:40:11.484492 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.484457 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:11.484492 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.484495 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrgj\" (UniqueName: \"kubernetes.io/projected/0de79225-c20d-4bba-991d-721309c70c42-kube-api-access-xjrgj\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:11.484680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.484513 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de79225-c20d-4bba-991d-721309c70c42-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:11.895901 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.895863 2575 generic.go:356] "Generic (PLEG): container finished" podID="0de79225-c20d-4bba-991d-721309c70c42" containerID="340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995" exitCode=0 Sep 15 21:40:11.896115 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.895917 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" Sep 15 21:40:11.896115 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.895959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" event={"ID":"0de79225-c20d-4bba-991d-721309c70c42","Type":"ContainerDied","Data":"340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995"} Sep 15 21:40:11.896115 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.896019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv" event={"ID":"0de79225-c20d-4bba-991d-721309c70c42","Type":"ContainerDied","Data":"91244f98763bbecfd877367ec8cbae5799286c579839c878eb40572f9ecf681d"} Sep 15 21:40:11.896115 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.896042 2575 scope.go:117] "RemoveContainer" containerID="340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995" Sep 15 21:40:11.907734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.906998 2575 scope.go:117] "RemoveContainer" containerID="340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995" Sep 15 21:40:11.907734 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:40:11.907637 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995\": container with ID starting with 340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995 not found: ID does not exist" containerID="340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995" Sep 15 21:40:11.907734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.907668 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995"} err="failed to get container status \"340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995\": rpc error: code = NotFound desc = could not find container \"340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995\": container with ID starting with 340bebeec7e57bc3984e110b71aece3821be156ac8583428bfe815f8de9df995 not found: ID does not exist" Sep 15 21:40:11.926561 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.926538 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv"] Sep 15 21:40:11.931361 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:11.931336 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba8d384-f6fcb68b-hvkpv"] Sep 15 21:40:13.147169 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:13.147141 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de79225-c20d-4bba-991d-721309c70c42" path="/var/lib/kubelet/pods/0de79225-c20d-4bba-991d-721309c70c42/volumes" Sep 15 21:40:16.354317 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.354283 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh"] Sep 15 21:40:16.354917 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.354677 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de79225-c20d-4bba-991d-721309c70c42" containerName="maas-api" Sep 15 21:40:16.354917 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.354688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de79225-c20d-4bba-991d-721309c70c42" containerName="maas-api" Sep 15 21:40:16.354917 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.354757 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de79225-c20d-4bba-991d-721309c70c42" containerName="maas-api" Sep 15 21:40:16.357826 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.357807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.360362 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.360340 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ee52fdef-serving-cert\"" Sep 15 21:40:16.370360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.370334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh"] Sep 15 21:40:16.428650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.428619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e955d-f254-43b5-8159-6b6818d33d9a-openshift-service-ca\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.428808 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.428660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-metrics-certs\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.428808 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.428747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-maas-api-tls\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.428897 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.428818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84cz\" (UniqueName: \"kubernetes.io/projected/927e955d-f254-43b5-8159-6b6818d33d9a-kube-api-access-j84cz\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.529273 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.529240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-maas-api-tls\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.529440 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.529288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84cz\" (UniqueName: \"kubernetes.io/projected/927e955d-f254-43b5-8159-6b6818d33d9a-kube-api-access-j84cz\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.529440 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.529333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e955d-f254-43b5-8159-6b6818d33d9a-openshift-service-ca\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.529440 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.529352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-metrics-certs\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.530030 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.530005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e955d-f254-43b5-8159-6b6818d33d9a-openshift-service-ca\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.531802 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.531779 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-metrics-certs\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.531965 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.531943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-maas-api-tls\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.537208 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.537185 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84cz\" (UniqueName: \"kubernetes.io/projected/927e955d-f254-43b5-8159-6b6818d33d9a-kube-api-access-j84cz\") pod \"maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.669132 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.669083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:16.942949 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:16.942925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh"] Sep 15 21:40:16.945025 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:40:16.944992 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927e955d_f254_43b5_8159_6b6818d33d9a.slice/crio-de35cadd8890e5c242573b3a17dccb73efc23aae87c8c24417eafb93fd777683 WatchSource:0}: Error finding container de35cadd8890e5c242573b3a17dccb73efc23aae87c8c24417eafb93fd777683: Status 404 returned error can't find the container with id de35cadd8890e5c242573b3a17dccb73efc23aae87c8c24417eafb93fd777683 Sep 15 21:40:17.931297 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:17.931251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" event={"ID":"927e955d-f254-43b5-8159-6b6818d33d9a","Type":"ContainerStarted","Data":"37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3"} Sep 15 21:40:17.931722 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:17.931314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" event={"ID":"927e955d-f254-43b5-8159-6b6818d33d9a","Type":"ContainerStarted","Data":"de35cadd8890e5c242573b3a17dccb73efc23aae87c8c24417eafb93fd777683"} Sep 15 21:40:17.931722 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:17.931382 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:17.948706 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:17.948658 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" podStartSLOduration=1.132994237 podStartE2EDuration="1.948642428s" totalImagesPullingTime="815.648191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:17.946863789 +0000 UTC m=+867.286343239" watchObservedRunningTime="2026-09-15 21:40:17.948642428 +0000 UTC m=+867.288121877" Sep 15 21:40:23.942358 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:23.942330 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:46.220947 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.220874 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq"] Sep 15 21:40:46.224582 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.224562 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.228148 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.228126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"kube-root-ca.crt\"" Sep 15 21:40:46.228234 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.228152 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"simulator-w0-360e43-kserve-self-signed-certs\"" Sep 15 21:40:46.228234 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.228131 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"openshift-service-ca.crt\"" Sep 15 21:40:46.228234 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.228137 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"default-dockercfg-nz88v\"" Sep 15 21:40:46.233634 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.233610 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq"] Sep 15 21:40:46.299628 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.299585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-model-cache\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.299789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.299655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-dshm\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.299789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.299702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfrh\" (UniqueName: \"kubernetes.io/projected/39bad9c6-6934-4522-b065-473d9891f551-kube-api-access-djfrh\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.299789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.299747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bad9c6-6934-4522-b065-473d9891f551-tls-certs\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.299789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.299788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-tmp-dir\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.300005 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.299837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-home\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.400692 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.400653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bad9c6-6934-4522-b065-473d9891f551-tls-certs\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.400932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.400715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-tmp-dir\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.400932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.400740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-home\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.400932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.400779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-model-cache\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.400932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.400827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-dshm\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.400932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.400865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfrh\" (UniqueName: \"kubernetes.io/projected/39bad9c6-6934-4522-b065-473d9891f551-kube-api-access-djfrh\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.401261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.401231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-tmp-dir\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.401261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.401250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-home\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.401432 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.401408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-model-cache\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.403609 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.403584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-dshm\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.403968 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.403949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bad9c6-6934-4522-b065-473d9891f551-tls-certs\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.418315 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.418289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfrh\" (UniqueName: \"kubernetes.io/projected/39bad9c6-6934-4522-b065-473d9891f551-kube-api-access-djfrh\") pod \"simulator-w0-360e43-kserve-6649cc64cb-q5rkq\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.554062 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.553985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:46.631823 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.631789 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh"] Sep 15 21:40:46.632309 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.632067 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" podUID="927e955d-f254-43b5-8159-6b6818d33d9a" containerName="maas-api" containerID="cri-o://37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3" gracePeriod=30 Sep 15 21:40:46.826800 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.826778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq"] Sep 15 21:40:46.829032 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:40:46.829004 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39bad9c6_6934_4522_b065_473d9891f551.slice/crio-223a426ac6c2d28a4b6236bc53c7639c20362e393e7494042ba72c59a9deb1a4 WatchSource:0}: Error finding container 223a426ac6c2d28a4b6236bc53c7639c20362e393e7494042ba72c59a9deb1a4: Status 404 returned error can't find the container with id 223a426ac6c2d28a4b6236bc53c7639c20362e393e7494042ba72c59a9deb1a4 Sep 15 21:40:46.898923 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.898901 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:46.905936 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.905862 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-maas-api-tls\") pod \"927e955d-f254-43b5-8159-6b6818d33d9a\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " Sep 15 21:40:46.906059 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.905972 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j84cz\" (UniqueName: \"kubernetes.io/projected/927e955d-f254-43b5-8159-6b6818d33d9a-kube-api-access-j84cz\") pod \"927e955d-f254-43b5-8159-6b6818d33d9a\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " Sep 15 21:40:46.906059 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.906032 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-metrics-certs\") pod \"927e955d-f254-43b5-8159-6b6818d33d9a\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " Sep 15 21:40:46.906200 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.906059 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e955d-f254-43b5-8159-6b6818d33d9a-openshift-service-ca\") pod \"927e955d-f254-43b5-8159-6b6818d33d9a\" (UID: \"927e955d-f254-43b5-8159-6b6818d33d9a\") " Sep 15 21:40:46.906968 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.906882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927e955d-f254-43b5-8159-6b6818d33d9a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "927e955d-f254-43b5-8159-6b6818d33d9a" (UID: "927e955d-f254-43b5-8159-6b6818d33d9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:40:46.908931 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.908896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927e955d-f254-43b5-8159-6b6818d33d9a-kube-api-access-j84cz" (OuterVolumeSpecName: "kube-api-access-j84cz") pod "927e955d-f254-43b5-8159-6b6818d33d9a" (UID: "927e955d-f254-43b5-8159-6b6818d33d9a"). InnerVolumeSpecName "kube-api-access-j84cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:40:46.909206 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.909182 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "927e955d-f254-43b5-8159-6b6818d33d9a" (UID: "927e955d-f254-43b5-8159-6b6818d33d9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:40:46.909366 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:46.909346 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "927e955d-f254-43b5-8159-6b6818d33d9a" (UID: "927e955d-f254-43b5-8159-6b6818d33d9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:40:47.007742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.007701 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:47.007742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.007729 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e955d-f254-43b5-8159-6b6818d33d9a-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:47.007742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.007739 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/927e955d-f254-43b5-8159-6b6818d33d9a-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:47.007742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.007747 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84cz\" (UniqueName: \"kubernetes.io/projected/927e955d-f254-43b5-8159-6b6818d33d9a-kube-api-access-j84cz\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:40:47.085111 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.085010 2575 generic.go:356] "Generic (PLEG): container finished" podID="927e955d-f254-43b5-8159-6b6818d33d9a" containerID="37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3" exitCode=0 Sep 15 21:40:47.085111 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.085060 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" Sep 15 21:40:47.085334 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.085091 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" event={"ID":"927e955d-f254-43b5-8159-6b6818d33d9a","Type":"ContainerDied","Data":"37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3"} Sep 15 21:40:47.085334 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.085177 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh" event={"ID":"927e955d-f254-43b5-8159-6b6818d33d9a","Type":"ContainerDied","Data":"de35cadd8890e5c242573b3a17dccb73efc23aae87c8c24417eafb93fd777683"} Sep 15 21:40:47.085334 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.085206 2575 scope.go:117] "RemoveContainer" containerID="37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3" Sep 15 21:40:47.087723 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.087701 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" event={"ID":"39bad9c6-6934-4522-b065-473d9891f551","Type":"ContainerStarted","Data":"33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b"} Sep 15 21:40:47.087848 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.087738 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" event={"ID":"39bad9c6-6934-4522-b065-473d9891f551","Type":"ContainerStarted","Data":"223a426ac6c2d28a4b6236bc53c7639c20362e393e7494042ba72c59a9deb1a4"} Sep 15 21:40:47.096614 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.096594 2575 scope.go:117] "RemoveContainer" containerID="37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3" Sep 15 21:40:47.096988 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:40:47.096967 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3\": container with ID starting with 37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3 not found: ID does not exist" containerID="37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3" Sep 15 21:40:47.097081 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.096995 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3"} err="failed to get container status \"37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3\": rpc error: code = NotFound desc = could not find container \"37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3\": container with ID starting with 37c780fea6857d830d47b9162ee2c4e0b0a0cc23b9912c09f0b54c9376d1adf3 not found: ID does not exist" Sep 15 21:40:47.110068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.110026 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" podStartSLOduration=1.110012148 podStartE2EDuration="1.110012148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:47.108214961 +0000 UTC m=+896.447694413" watchObservedRunningTime="2026-09-15 21:40:47.110012148 +0000 UTC m=+896.449491647" Sep 15 21:40:47.123073 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.123051 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh"] Sep 15 21:40:47.126186 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.126160 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ee52fdef-58877546bb-6m4gh"] Sep 15 21:40:47.146981 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:47.146954 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="927e955d-f254-43b5-8159-6b6818d33d9a" path="/var/lib/kubelet/pods/927e955d-f254-43b5-8159-6b6818d33d9a/volumes" Sep 15 21:40:52.135833 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.135794 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7"] Sep 15 21:40:52.136382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.136299 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927e955d-f254-43b5-8159-6b6818d33d9a" containerName="maas-api" Sep 15 21:40:52.136382 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.136316 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="927e955d-f254-43b5-8159-6b6818d33d9a" containerName="maas-api" Sep 15 21:40:52.136499 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.136387 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="927e955d-f254-43b5-8159-6b6818d33d9a" containerName="maas-api" Sep 15 21:40:52.142771 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.142732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.144648 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.144411 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7"] Sep 15 21:40:52.145303 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.145281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8316f91a-serving-cert\"" Sep 15 21:40:52.254796 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.254751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-maas-api-tls\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.254988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.254803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpb9\" (UniqueName: \"kubernetes.io/projected/c3776e23-1418-4bfd-b95e-25d934ae0eb9-kube-api-access-fwpb9\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.254988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.254887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-metrics-certs\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.255138 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.255037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3776e23-1418-4bfd-b95e-25d934ae0eb9-openshift-service-ca\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.356191 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.356160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-maas-api-tls\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.356191 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.356204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpb9\" (UniqueName: \"kubernetes.io/projected/c3776e23-1418-4bfd-b95e-25d934ae0eb9-kube-api-access-fwpb9\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.356451 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.356231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-metrics-certs\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.356451 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.356307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3776e23-1418-4bfd-b95e-25d934ae0eb9-openshift-service-ca\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.357068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.357041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3776e23-1418-4bfd-b95e-25d934ae0eb9-openshift-service-ca\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.358962 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.358934 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-metrics-certs\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.359061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.358966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-maas-api-tls\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.364041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.364018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpb9\" (UniqueName: \"kubernetes.io/projected/c3776e23-1418-4bfd-b95e-25d934ae0eb9-kube-api-access-fwpb9\") pod \"maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.457131 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.457064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:52.752330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:52.752285 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7"] Sep 15 21:40:53.124862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:53.124832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" event={"ID":"c3776e23-1418-4bfd-b95e-25d934ae0eb9","Type":"ContainerStarted","Data":"182eb0938d1f963adcfd495c4dbf18206a78dc7650d732948088e2fb41400f91"} Sep 15 21:40:54.132384 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:54.132346 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" event={"ID":"c3776e23-1418-4bfd-b95e-25d934ae0eb9","Type":"ContainerStarted","Data":"fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d"} Sep 15 21:40:54.132809 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:54.132462 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:40:54.149488 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:54.149435 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" podStartSLOduration=1.8088573289999998 podStartE2EDuration="2.149418873s" totalImagesPullingTime="340.561544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:54.147762934 +0000 UTC m=+903.487242387" watchObservedRunningTime="2026-09-15 21:40:54.149418873 +0000 UTC m=+903.488898324" Sep 15 21:40:56.554281 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:56.554245 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:56.554750 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:56.554309 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:56.567048 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:56.567018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:40:57.159274 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:40:57.159247 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:41:00.143337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:00.143310 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:41:21.670895 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:21.670856 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7"] Sep 15 21:41:21.671334 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:21.671137 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" podUID="c3776e23-1418-4bfd-b95e-25d934ae0eb9" containerName="maas-api" containerID="cri-o://fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d" gracePeriod=30 Sep 15 21:41:21.932720 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:21.932663 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:41:22.046659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.046624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-metrics-certs\") pod \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " Sep 15 21:41:22.046659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.046662 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwpb9\" (UniqueName: \"kubernetes.io/projected/c3776e23-1418-4bfd-b95e-25d934ae0eb9-kube-api-access-fwpb9\") pod \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " Sep 15 21:41:22.046898 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.046679 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-maas-api-tls\") pod \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " Sep 15 21:41:22.046898 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.046720 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3776e23-1418-4bfd-b95e-25d934ae0eb9-openshift-service-ca\") pod \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\" (UID: \"c3776e23-1418-4bfd-b95e-25d934ae0eb9\") " Sep 15 21:41:22.047153 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.047125 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3776e23-1418-4bfd-b95e-25d934ae0eb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3776e23-1418-4bfd-b95e-25d934ae0eb9" (UID: "c3776e23-1418-4bfd-b95e-25d934ae0eb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:41:22.048908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.048886 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3776e23-1418-4bfd-b95e-25d934ae0eb9" (UID: "c3776e23-1418-4bfd-b95e-25d934ae0eb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:41:22.049007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.048952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3776e23-1418-4bfd-b95e-25d934ae0eb9-kube-api-access-fwpb9" (OuterVolumeSpecName: "kube-api-access-fwpb9") pod "c3776e23-1418-4bfd-b95e-25d934ae0eb9" (UID: "c3776e23-1418-4bfd-b95e-25d934ae0eb9"). InnerVolumeSpecName "kube-api-access-fwpb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:41:22.049007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.048975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3776e23-1418-4bfd-b95e-25d934ae0eb9" (UID: "c3776e23-1418-4bfd-b95e-25d934ae0eb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:41:22.147866 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.147834 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:41:22.147866 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.147860 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwpb9\" (UniqueName: \"kubernetes.io/projected/c3776e23-1418-4bfd-b95e-25d934ae0eb9-kube-api-access-fwpb9\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:41:22.147866 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.147871 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3776e23-1418-4bfd-b95e-25d934ae0eb9-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:41:22.148120 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.147879 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3776e23-1418-4bfd-b95e-25d934ae0eb9-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:41:22.284018 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.283927 2575 generic.go:356] "Generic (PLEG): container finished" podID="c3776e23-1418-4bfd-b95e-25d934ae0eb9" containerID="fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d" exitCode=0 Sep 15 21:41:22.284018 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.283988 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" Sep 15 21:41:22.284386 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.284019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" event={"ID":"c3776e23-1418-4bfd-b95e-25d934ae0eb9","Type":"ContainerDied","Data":"fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d"} Sep 15 21:41:22.284386 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.284078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7" event={"ID":"c3776e23-1418-4bfd-b95e-25d934ae0eb9","Type":"ContainerDied","Data":"182eb0938d1f963adcfd495c4dbf18206a78dc7650d732948088e2fb41400f91"} Sep 15 21:41:22.284386 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.284130 2575 scope.go:117] "RemoveContainer" containerID="fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d" Sep 15 21:41:22.293573 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.293558 2575 scope.go:117] "RemoveContainer" containerID="fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d" Sep 15 21:41:22.293860 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:41:22.293842 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d\": container with ID starting with fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d not found: ID does not exist" containerID="fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d" Sep 15 21:41:22.293927 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.293870 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d"} err="failed to get container status \"fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d\": rpc error: code = NotFound desc = could not find container \"fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d\": container with ID starting with fffc30d21be5d486c245cb50a85f0656615b70df3235356a882c9d9d206d212d not found: ID does not exist" Sep 15 21:41:22.308041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.308016 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7"] Sep 15 21:41:22.317198 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:22.317170 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8316f91a-8fc84c78-qlhv7"] Sep 15 21:41:23.147668 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:23.147622 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3776e23-1418-4bfd-b95e-25d934ae0eb9" path="/var/lib/kubelet/pods/c3776e23-1418-4bfd-b95e-25d934ae0eb9/volumes" Sep 15 21:41:31.259158 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.259120 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv"] Sep 15 21:41:31.261513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.259631 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3776e23-1418-4bfd-b95e-25d934ae0eb9" containerName="maas-api" Sep 15 21:41:31.261513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.259646 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3776e23-1418-4bfd-b95e-25d934ae0eb9" containerName="maas-api" Sep 15 21:41:31.261513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.259711 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3776e23-1418-4bfd-b95e-25d934ae0eb9" containerName="maas-api" Sep 15 21:41:31.262586 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.262572 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.265030 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.265010 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8ce29e8a-serving-cert\"" Sep 15 21:41:31.271976 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.271957 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv"] Sep 15 21:41:31.328618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.328586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-metrics-certs\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.328814 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.328682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.328814 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.328715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-openshift-service-ca\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.328814 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.328749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7pw\" (UniqueName: \"kubernetes.io/projected/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-kube-api-access-4n7pw\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.429517 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.429478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-metrics-certs\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.429731 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.429572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.429731 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.429596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-openshift-service-ca\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.429731 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.429626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7pw\" (UniqueName: \"kubernetes.io/projected/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-kube-api-access-4n7pw\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.429731 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:41:31.429670 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-serving-cert: secret "maas-api-e2e-derive-8ce29e8a-serving-cert" not found Sep 15 21:41:31.429911 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:41:31.429763 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls podName:ae56af1f-22ea-4e7a-8570-2685eaa7a43f nodeName:}" failed. No retries permitted until 2026-09-15 21:41:31.929742155 +0000 UTC m=+941.269221593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls") pod "maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" (UID: "ae56af1f-22ea-4e7a-8570-2685eaa7a43f") : secret "maas-api-e2e-derive-8ce29e8a-serving-cert" not found Sep 15 21:41:31.430486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.430459 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-openshift-service-ca\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.432128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.432090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-metrics-certs\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.437943 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.437924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7pw\" (UniqueName: \"kubernetes.io/projected/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-kube-api-access-4n7pw\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.934493 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.934456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:31.937129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:31.937053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls\") pod \"maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:32.176178 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:32.176142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:32.451336 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:32.451304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv"] Sep 15 21:41:32.454583 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:41:32.454545 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae56af1f_22ea_4e7a_8570_2685eaa7a43f.slice/crio-a902d376e4d59a9ad3a34989802e77c17bc6d703d9d51313a0a18d701ab18d12 WatchSource:0}: Error finding container a902d376e4d59a9ad3a34989802e77c17bc6d703d9d51313a0a18d701ab18d12: Status 404 returned error can't find the container with id a902d376e4d59a9ad3a34989802e77c17bc6d703d9d51313a0a18d701ab18d12 Sep 15 21:41:33.345306 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:33.345268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" event={"ID":"ae56af1f-22ea-4e7a-8570-2685eaa7a43f","Type":"ContainerStarted","Data":"3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752"} Sep 15 21:41:33.345467 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:33.345332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" event={"ID":"ae56af1f-22ea-4e7a-8570-2685eaa7a43f","Type":"ContainerStarted","Data":"a902d376e4d59a9ad3a34989802e77c17bc6d703d9d51313a0a18d701ab18d12"} Sep 15 21:41:33.345467 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:33.345412 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:33.366793 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:33.366689 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" podStartSLOduration=1.710562044 podStartE2EDuration="2.366670307s" totalImagesPullingTime="656.108263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:33.36412344 +0000 UTC m=+942.703602882" watchObservedRunningTime="2026-09-15 21:41:33.366670307 +0000 UTC m=+942.706149757" Sep 15 21:41:39.355350 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:39.355317 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:41:40.919813 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.919768 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds"] Sep 15 21:41:40.923732 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.923709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:40.926061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.926045 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5416b7\"/\"openshift-service-ca.crt\"" Sep 15 21:41:40.927132 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.927089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5416b7\"/\"default-dockercfg-d929t\"" Sep 15 21:41:40.927132 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.927126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5416b7\"/\"kube-root-ca.crt\"" Sep 15 21:41:40.927256 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.927131 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5416b7\"/\"autores-model-5416b7-kserve-self-signed-certs\"" Sep 15 21:41:40.930364 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:40.930342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds"] Sep 15 21:41:41.017687 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.017659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84eff205-e40f-4857-b817-7fc2d6acb8cd-tls-certs\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.017875 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.017705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbw5\" (UniqueName: \"kubernetes.io/projected/84eff205-e40f-4857-b817-7fc2d6acb8cd-kube-api-access-rqbw5\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.017875 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.017728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-tmp-dir\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.017875 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.017755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-home\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.017875 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.017864 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-dshm\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.018094 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.017902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-model-cache\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.118751 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.118719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84eff205-e40f-4857-b817-7fc2d6acb8cd-tls-certs\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.118908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.118776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbw5\" (UniqueName: \"kubernetes.io/projected/84eff205-e40f-4857-b817-7fc2d6acb8cd-kube-api-access-rqbw5\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.118908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.118811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-tmp-dir\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.118908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.118835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-home\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.118908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.118873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-dshm\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.118908 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.118895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-model-cache\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.119320 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.119289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-tmp-dir\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.119320 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.119304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-model-cache\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.119504 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.119347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-home\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.121267 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.121245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-dshm\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.121550 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.121533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84eff205-e40f-4857-b817-7fc2d6acb8cd-tls-certs\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.126504 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.126487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbw5\" (UniqueName: \"kubernetes.io/projected/84eff205-e40f-4857-b817-7fc2d6acb8cd-kube-api-access-rqbw5\") pod \"autores-model-5416b7-kserve-f9778689f-rcjds\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.234235 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.234152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:41.507847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:41.507825 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds"] Sep 15 21:41:41.510076 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:41:41.510050 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84eff205_e40f_4857_b817_7fc2d6acb8cd.slice/crio-77d7dbd42a59c26160d542c8c0b4469914c071262eb212da7ed18331c10407ec WatchSource:0}: Error finding container 77d7dbd42a59c26160d542c8c0b4469914c071262eb212da7ed18331c10407ec: Status 404 returned error can't find the container with id 77d7dbd42a59c26160d542c8c0b4469914c071262eb212da7ed18331c10407ec Sep 15 21:41:42.395848 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:42.395816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" event={"ID":"84eff205-e40f-4857-b817-7fc2d6acb8cd","Type":"ContainerStarted","Data":"f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726"} Sep 15 21:41:42.396271 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:42.395860 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" event={"ID":"84eff205-e40f-4857-b817-7fc2d6acb8cd","Type":"ContainerStarted","Data":"77d7dbd42a59c26160d542c8c0b4469914c071262eb212da7ed18331c10407ec"} Sep 15 21:41:42.415546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:42.415500 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" podStartSLOduration=2.41548586 podStartE2EDuration="2.41548586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:42.413624853 +0000 UTC m=+951.753104304" watchObservedRunningTime="2026-09-15 21:41:42.41548586 +0000 UTC m=+951.754965311" Sep 15 21:41:51.234708 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:51.234675 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:51.235056 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:51.234727 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:51.247432 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:51.247406 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:41:51.456221 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:41:51.456197 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:42:09.052920 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:09.052880 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds"] Sep 15 21:42:13.882604 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:13.882509 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv"] Sep 15 21:42:13.883050 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:13.882792 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" podUID="ae56af1f-22ea-4e7a-8570-2685eaa7a43f" containerName="maas-api" containerID="cri-o://3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752" gracePeriod=30 Sep 15 21:42:14.142607 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.142553 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:42:14.319377 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.319345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n7pw\" (UniqueName: \"kubernetes.io/projected/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-kube-api-access-4n7pw\") pod \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " Sep 15 21:42:14.319536 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.319408 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls\") pod \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " Sep 15 21:42:14.319536 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.319464 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-metrics-certs\") pod \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " Sep 15 21:42:14.319536 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.319483 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-openshift-service-ca\") pod \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\" (UID: \"ae56af1f-22ea-4e7a-8570-2685eaa7a43f\") " Sep 15 21:42:14.319854 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.319826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae56af1f-22ea-4e7a-8570-2685eaa7a43f" (UID: "ae56af1f-22ea-4e7a-8570-2685eaa7a43f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:42:14.321587 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.321560 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-kube-api-access-4n7pw" (OuterVolumeSpecName: "kube-api-access-4n7pw") pod "ae56af1f-22ea-4e7a-8570-2685eaa7a43f" (UID: "ae56af1f-22ea-4e7a-8570-2685eaa7a43f"). InnerVolumeSpecName "kube-api-access-4n7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:42:14.321587 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.321569 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ae56af1f-22ea-4e7a-8570-2685eaa7a43f" (UID: "ae56af1f-22ea-4e7a-8570-2685eaa7a43f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:14.321710 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.321658 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae56af1f-22ea-4e7a-8570-2685eaa7a43f" (UID: "ae56af1f-22ea-4e7a-8570-2685eaa7a43f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:14.420368 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.420335 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:14.420368 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.420367 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:14.420559 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.420382 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7pw\" (UniqueName: \"kubernetes.io/projected/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-kube-api-access-4n7pw\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:14.420559 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.420393 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae56af1f-22ea-4e7a-8570-2685eaa7a43f-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:14.566015 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.565981 2575 generic.go:356] "Generic (PLEG): container finished" podID="ae56af1f-22ea-4e7a-8570-2685eaa7a43f" containerID="3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752" exitCode=0 Sep 15 21:42:14.566206 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.566029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" event={"ID":"ae56af1f-22ea-4e7a-8570-2685eaa7a43f","Type":"ContainerDied","Data":"3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752"} Sep 15 21:42:14.566206 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.566067 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" event={"ID":"ae56af1f-22ea-4e7a-8570-2685eaa7a43f","Type":"ContainerDied","Data":"a902d376e4d59a9ad3a34989802e77c17bc6d703d9d51313a0a18d701ab18d12"} Sep 15 21:42:14.566206 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.566089 2575 scope.go:117] "RemoveContainer" containerID="3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752" Sep 15 21:42:14.566206 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.566029 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv" Sep 15 21:42:14.576495 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.576454 2575 scope.go:117] "RemoveContainer" containerID="3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752" Sep 15 21:42:14.576805 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:42:14.576777 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752\": container with ID starting with 3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752 not found: ID does not exist" containerID="3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752" Sep 15 21:42:14.576924 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.576819 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752"} err="failed to get container status \"3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752\": rpc error: code = NotFound desc = could not find container \"3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752\": container with ID starting with 3205a16390968249d6617983c485d44175471e596e8eebd2931f61bd43b1f752 not found: ID does not exist" Sep 15 21:42:14.588511 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.588483 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv"] Sep 15 21:42:14.592694 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:14.592670 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8ce29e8a-584b6ccc85-9x7mv"] Sep 15 21:42:15.147643 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:15.147608 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae56af1f-22ea-4e7a-8570-2685eaa7a43f" path="/var/lib/kubelet/pods/ae56af1f-22ea-4e7a-8570-2685eaa7a43f/volumes" Sep 15 21:42:19.632457 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.632423 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz"] Sep 15 21:42:19.632942 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.632673 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" podUID="86a7d18b-19aa-47da-915e-d8c03b812a0f" containerName="maas-api" containerID="cri-o://e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb" gracePeriod=30 Sep 15 21:42:19.901470 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.901446 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:42:19.970535 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.970504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vn9b\" (UniqueName: \"kubernetes.io/projected/86a7d18b-19aa-47da-915e-d8c03b812a0f-kube-api-access-2vn9b\") pod \"86a7d18b-19aa-47da-915e-d8c03b812a0f\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " Sep 15 21:42:19.970535 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.970536 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7d18b-19aa-47da-915e-d8c03b812a0f-openshift-service-ca\") pod \"86a7d18b-19aa-47da-915e-d8c03b812a0f\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " Sep 15 21:42:19.970751 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.970601 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-maas-api-tls\") pod \"86a7d18b-19aa-47da-915e-d8c03b812a0f\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " Sep 15 21:42:19.970751 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.970674 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-metrics-certs\") pod \"86a7d18b-19aa-47da-915e-d8c03b812a0f\" (UID: \"86a7d18b-19aa-47da-915e-d8c03b812a0f\") " Sep 15 21:42:19.970923 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.970891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a7d18b-19aa-47da-915e-d8c03b812a0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86a7d18b-19aa-47da-915e-d8c03b812a0f" (UID: "86a7d18b-19aa-47da-915e-d8c03b812a0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:42:19.973031 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.972993 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "86a7d18b-19aa-47da-915e-d8c03b812a0f" (UID: "86a7d18b-19aa-47da-915e-d8c03b812a0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:19.973031 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.973007 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a7d18b-19aa-47da-915e-d8c03b812a0f-kube-api-access-2vn9b" (OuterVolumeSpecName: "kube-api-access-2vn9b") pod "86a7d18b-19aa-47da-915e-d8c03b812a0f" (UID: "86a7d18b-19aa-47da-915e-d8c03b812a0f"). InnerVolumeSpecName "kube-api-access-2vn9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:42:19.973246 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:19.973202 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86a7d18b-19aa-47da-915e-d8c03b812a0f" (UID: "86a7d18b-19aa-47da-915e-d8c03b812a0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:20.071899 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.071869 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:20.071899 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.071893 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7d18b-19aa-47da-915e-d8c03b812a0f-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:20.071899 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.071904 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vn9b\" (UniqueName: \"kubernetes.io/projected/86a7d18b-19aa-47da-915e-d8c03b812a0f-kube-api-access-2vn9b\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:20.072158 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.071915 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7d18b-19aa-47da-915e-d8c03b812a0f-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:20.606989 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.606951 2575 generic.go:356] "Generic (PLEG): container finished" podID="86a7d18b-19aa-47da-915e-d8c03b812a0f" containerID="e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb" exitCode=0 Sep 15 21:42:20.607215 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.606999 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" Sep 15 21:42:20.607215 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.607042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" event={"ID":"86a7d18b-19aa-47da-915e-d8c03b812a0f","Type":"ContainerDied","Data":"e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb"} Sep 15 21:42:20.607215 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.607140 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz" event={"ID":"86a7d18b-19aa-47da-915e-d8c03b812a0f","Type":"ContainerDied","Data":"16d550dabadb3c8975533a09a81f94e81936a57dcfe6f431470586138d5015ec"} Sep 15 21:42:20.607215 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.607173 2575 scope.go:117] "RemoveContainer" containerID="e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb" Sep 15 21:42:20.617373 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.617356 2575 scope.go:117] "RemoveContainer" containerID="e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb" Sep 15 21:42:20.617640 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:42:20.617620 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb\": container with ID starting with e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb not found: ID does not exist" containerID="e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb" Sep 15 21:42:20.617695 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.617649 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb"} err="failed to get container status \"e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb\": rpc error: code = NotFound desc = could not find container \"e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb\": container with ID starting with e72c40562c73011089db03a38951ee73aa490b71fcbaa153774e030b8c9b2bcb not found: ID does not exist" Sep 15 21:42:20.627585 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.627564 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz"] Sep 15 21:42:20.631424 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:20.631400 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5416b7-cb55f4c95-9npsz"] Sep 15 21:42:21.147292 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:21.147263 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a7d18b-19aa-47da-915e-d8c03b812a0f" path="/var/lib/kubelet/pods/86a7d18b-19aa-47da-915e-d8c03b812a0f/volumes" Sep 15 21:42:24.073284 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.073228 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" podUID="84eff205-e40f-4857-b817-7fc2d6acb8cd" containerName="main" containerID="cri-o://f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726" gracePeriod=45 Sep 15 21:42:24.345226 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.345203 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:42:24.414681 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414652 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqbw5\" (UniqueName: \"kubernetes.io/projected/84eff205-e40f-4857-b817-7fc2d6acb8cd-kube-api-access-rqbw5\") pod \"84eff205-e40f-4857-b817-7fc2d6acb8cd\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " Sep 15 21:42:24.414835 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414689 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-home\") pod \"84eff205-e40f-4857-b817-7fc2d6acb8cd\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " Sep 15 21:42:24.414835 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414716 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-model-cache\") pod \"84eff205-e40f-4857-b817-7fc2d6acb8cd\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " Sep 15 21:42:24.414835 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414825 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-tmp-dir\") pod \"84eff205-e40f-4857-b817-7fc2d6acb8cd\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " Sep 15 21:42:24.415002 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84eff205-e40f-4857-b817-7fc2d6acb8cd-tls-certs\") pod \"84eff205-e40f-4857-b817-7fc2d6acb8cd\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " Sep 15 21:42:24.415002 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414878 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-dshm\") pod \"84eff205-e40f-4857-b817-7fc2d6acb8cd\" (UID: \"84eff205-e40f-4857-b817-7fc2d6acb8cd\") " Sep 15 21:42:24.415002 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414940 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-home" (OuterVolumeSpecName: "home") pod "84eff205-e40f-4857-b817-7fc2d6acb8cd" (UID: "84eff205-e40f-4857-b817-7fc2d6acb8cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:24.415002 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.414965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "84eff205-e40f-4857-b817-7fc2d6acb8cd" (UID: "84eff205-e40f-4857-b817-7fc2d6acb8cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:24.415259 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.415059 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84eff205-e40f-4857-b817-7fc2d6acb8cd" (UID: "84eff205-e40f-4857-b817-7fc2d6acb8cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:24.415259 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.415249 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:24.415338 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.415266 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:24.415338 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.415278 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:24.417120 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.417057 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-dshm" (OuterVolumeSpecName: "dshm") pod "84eff205-e40f-4857-b817-7fc2d6acb8cd" (UID: "84eff205-e40f-4857-b817-7fc2d6acb8cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:42:24.417221 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.417146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84eff205-e40f-4857-b817-7fc2d6acb8cd-kube-api-access-rqbw5" (OuterVolumeSpecName: "kube-api-access-rqbw5") pod "84eff205-e40f-4857-b817-7fc2d6acb8cd" (UID: "84eff205-e40f-4857-b817-7fc2d6acb8cd"). InnerVolumeSpecName "kube-api-access-rqbw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:42:24.417221 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.417182 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84eff205-e40f-4857-b817-7fc2d6acb8cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84eff205-e40f-4857-b817-7fc2d6acb8cd" (UID: "84eff205-e40f-4857-b817-7fc2d6acb8cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:24.516512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.516479 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbw5\" (UniqueName: \"kubernetes.io/projected/84eff205-e40f-4857-b817-7fc2d6acb8cd-kube-api-access-rqbw5\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:24.516512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.516509 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84eff205-e40f-4857-b817-7fc2d6acb8cd-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:24.516512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.516520 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84eff205-e40f-4857-b817-7fc2d6acb8cd-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:42:24.629640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.629558 2575 generic.go:356] "Generic (PLEG): container finished" podID="84eff205-e40f-4857-b817-7fc2d6acb8cd" containerID="f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726" exitCode=0 Sep 15 21:42:24.629640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.629599 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" Sep 15 21:42:24.629819 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.629602 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" event={"ID":"84eff205-e40f-4857-b817-7fc2d6acb8cd","Type":"ContainerDied","Data":"f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726"} Sep 15 21:42:24.629819 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.629694 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds" event={"ID":"84eff205-e40f-4857-b817-7fc2d6acb8cd","Type":"ContainerDied","Data":"77d7dbd42a59c26160d542c8c0b4469914c071262eb212da7ed18331c10407ec"} Sep 15 21:42:24.629819 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.629710 2575 scope.go:117] "RemoveContainer" containerID="f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726" Sep 15 21:42:24.639603 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.639585 2575 scope.go:117] "RemoveContainer" containerID="f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726" Sep 15 21:42:24.639866 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:42:24.639849 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726\": container with ID starting with f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726 not found: ID does not exist" containerID="f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726" Sep 15 21:42:24.639915 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.639873 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726"} err="failed to get container status \"f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726\": rpc error: code = NotFound desc = could not find container \"f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726\": container with ID starting with f0599f5ad0354a317d136d926d9fcd3a0a95e88cc3c4f37b494bb7844b7d3726 not found: ID does not exist" Sep 15 21:42:24.649744 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.649721 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds"] Sep 15 21:42:24.653735 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:24.653711 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5416b7/autores-model-5416b7-kserve-f9778689f-rcjds"] Sep 15 21:42:25.147849 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:25.147812 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84eff205-e40f-4857-b817-7fc2d6acb8cd" path="/var/lib/kubelet/pods/84eff205-e40f-4857-b817-7fc2d6acb8cd/volumes" Sep 15 21:42:33.626920 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.626883 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m"] Sep 15 21:42:33.627513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627496 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84eff205-e40f-4857-b817-7fc2d6acb8cd" containerName="main" Sep 15 21:42:33.627551 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627516 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84eff205-e40f-4857-b817-7fc2d6acb8cd" containerName="main" Sep 15 21:42:33.627551 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627534 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae56af1f-22ea-4e7a-8570-2685eaa7a43f" containerName="maas-api" Sep 15 21:42:33.627551 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627542 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae56af1f-22ea-4e7a-8570-2685eaa7a43f" containerName="maas-api" Sep 15 21:42:33.627655 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627579 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a7d18b-19aa-47da-915e-d8c03b812a0f" containerName="maas-api" Sep 15 21:42:33.627655 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627588 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a7d18b-19aa-47da-915e-d8c03b812a0f" containerName="maas-api" Sep 15 21:42:33.627719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627670 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a7d18b-19aa-47da-915e-d8c03b812a0f" containerName="maas-api" Sep 15 21:42:33.627719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627683 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84eff205-e40f-4857-b817-7fc2d6acb8cd" containerName="main" Sep 15 21:42:33.627719 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.627693 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae56af1f-22ea-4e7a-8570-2685eaa7a43f" containerName="maas-api" Sep 15 21:42:33.632487 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.632460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.634871 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.634842 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4a3a26-serving-cert\"" Sep 15 21:42:33.641755 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.641296 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m"] Sep 15 21:42:33.698560 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.698517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-metrics-certs\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.698768 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.698600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkrm\" (UniqueName: \"kubernetes.io/projected/a2f7745b-1a56-42f8-b62b-5965f97cf733-kube-api-access-5dkrm\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.698768 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.698637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.698768 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.698685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f7745b-1a56-42f8-b62b-5965f97cf733-openshift-service-ca\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.800054 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.800012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f7745b-1a56-42f8-b62b-5965f97cf733-openshift-service-ca\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.800340 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.800130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-metrics-certs\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.800340 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.800203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkrm\" (UniqueName: \"kubernetes.io/projected/a2f7745b-1a56-42f8-b62b-5965f97cf733-kube-api-access-5dkrm\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.800340 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.800241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.800526 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:42:33.800357 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-serving-cert: secret "maas-api-e2e-explicit-4a3a26-serving-cert" not found Sep 15 21:42:33.800526 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:42:33.800439 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls podName:a2f7745b-1a56-42f8-b62b-5965f97cf733 nodeName:}" failed. No retries permitted until 2026-09-15 21:42:34.30041825 +0000 UTC m=+1003.639897696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls") pod "maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" (UID: "a2f7745b-1a56-42f8-b62b-5965f97cf733") : secret "maas-api-e2e-explicit-4a3a26-serving-cert" not found Sep 15 21:42:33.800809 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.800788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f7745b-1a56-42f8-b62b-5965f97cf733-openshift-service-ca\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.802870 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.802850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-metrics-certs\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:33.808607 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:33.808586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkrm\" (UniqueName: \"kubernetes.io/projected/a2f7745b-1a56-42f8-b62b-5965f97cf733-kube-api-access-5dkrm\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:34.305646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:34.305606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:34.308208 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:34.308184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls\") pod \"maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:34.545958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:34.545914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:34.827428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:34.827397 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m"] Sep 15 21:42:34.829593 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:42:34.829567 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f7745b_1a56_42f8_b62b_5965f97cf733.slice/crio-cc245257e5c6673923b6f26e9d507c3f39dd955e724785ac958e6e6324e98ae8 WatchSource:0}: Error finding container cc245257e5c6673923b6f26e9d507c3f39dd955e724785ac958e6e6324e98ae8: Status 404 returned error can't find the container with id cc245257e5c6673923b6f26e9d507c3f39dd955e724785ac958e6e6324e98ae8 Sep 15 21:42:34.831316 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:34.831297 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:42:35.695843 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:35.695806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" event={"ID":"a2f7745b-1a56-42f8-b62b-5965f97cf733","Type":"ContainerStarted","Data":"6ae0e78bd2c40562cd9e34a9d9222f25e1e5eba865d2ebc1b98649a3162f1844"} Sep 15 21:42:35.696031 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:35.695864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" event={"ID":"a2f7745b-1a56-42f8-b62b-5965f97cf733","Type":"ContainerStarted","Data":"cc245257e5c6673923b6f26e9d507c3f39dd955e724785ac958e6e6324e98ae8"} Sep 15 21:42:35.696031 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:35.695936 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:42:35.711694 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:35.711632 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" podStartSLOduration=2.298476577 podStartE2EDuration="2.711610537s" totalImagesPullingTime="413.13396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:35.710926403 +0000 UTC m=+1005.050405853" watchObservedRunningTime="2026-09-15 21:42:35.711610537 +0000 UTC m=+1005.051089989" Sep 15 21:42:41.706259 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:42:41.706229 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:43:05.918782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:05.918743 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9"] Sep 15 21:43:05.923918 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:05.923894 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:05.926332 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:05.926306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"premium-w0-360e43-kserve-self-signed-certs\"" Sep 15 21:43:05.933944 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:05.933904 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9"] Sep 15 21:43:06.000349 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.000315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-model-cache\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.000537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.000400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-home\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.000537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.000417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tmp-dir\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.000537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.000471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tls-certs\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.000537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.000501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dm7v\" (UniqueName: \"kubernetes.io/projected/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-kube-api-access-4dm7v\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.000537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.000522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-dshm\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.101632 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.101588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tls-certs\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.101632 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.101627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dm7v\" (UniqueName: \"kubernetes.io/projected/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-kube-api-access-4dm7v\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.101862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.101652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-dshm\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.101862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.101668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-model-cache\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.101862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.101720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-home\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.101862 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.101735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tmp-dir\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.102230 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.102205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-home\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.102321 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.102215 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-model-cache\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.102321 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.102260 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tmp-dir\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.104070 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.104048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-dshm\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.104326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.104308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tls-certs\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.109700 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.109680 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dm7v\" (UniqueName: \"kubernetes.io/projected/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-kube-api-access-4dm7v\") pod \"premium-w0-360e43-kserve-7fdbbb6869-6kgk9\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.235362 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.235253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:06.509826 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.509743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9"] Sep 15 21:43:06.513065 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:43:06.513035 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fec278a_d859_42d9_bdf9_d9fdcb5e82d8.slice/crio-691da5711bec715d40ce9be2a9fc9125a8659dd9a53d79c7e8c454b15393e7f7 WatchSource:0}: Error finding container 691da5711bec715d40ce9be2a9fc9125a8659dd9a53d79c7e8c454b15393e7f7: Status 404 returned error can't find the container with id 691da5711bec715d40ce9be2a9fc9125a8659dd9a53d79c7e8c454b15393e7f7 Sep 15 21:43:06.855281 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.855180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" event={"ID":"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8","Type":"ContainerStarted","Data":"166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8"} Sep 15 21:43:06.855281 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.855228 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" event={"ID":"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8","Type":"ContainerStarted","Data":"691da5711bec715d40ce9be2a9fc9125a8659dd9a53d79c7e8c454b15393e7f7"} Sep 15 21:43:06.874360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:06.874309 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" podStartSLOduration=1.87429417 podStartE2EDuration="1.87429417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:06.871653512 +0000 UTC m=+1036.211132974" watchObservedRunningTime="2026-09-15 21:43:06.87429417 +0000 UTC m=+1036.213773620" Sep 15 21:43:16.235696 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:16.235651 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:16.236242 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:16.235746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:16.248505 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:16.248474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:16.917281 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:16.917251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:43:21.340686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.340638 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z"] Sep 15 21:43:21.345061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.345038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.347486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.347465 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"openshift-service-ca.crt\"" Sep 15 21:43:21.347631 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.347465 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"kube-root-ca.crt\"" Sep 15 21:43:21.348431 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.348409 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"premium-w4-56e305-kserve-self-signed-certs\"" Sep 15 21:43:21.348553 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.348481 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"default-dockercfg-tfg85\"" Sep 15 21:43:21.353603 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.353581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z"] Sep 15 21:43:21.447514 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.447475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-dshm\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.447514 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.447518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa219405-3826-46fe-8c69-ea3042af755c-tls-certs\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.447742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.447539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4xm\" (UniqueName: \"kubernetes.io/projected/fa219405-3826-46fe-8c69-ea3042af755c-kube-api-access-fz4xm\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.447742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.447652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-model-cache\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.447742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.447684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-tmp-dir\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.447742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.447715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-home\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548368 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-dshm\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548368 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa219405-3826-46fe-8c69-ea3042af755c-tls-certs\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548616 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4xm\" (UniqueName: \"kubernetes.io/projected/fa219405-3826-46fe-8c69-ea3042af755c-kube-api-access-fz4xm\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548616 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-model-cache\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548616 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-tmp-dir\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548616 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-home\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-tmp-dir\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.548988 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-home\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.549029 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.548981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-model-cache\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.550982 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.550959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-dshm\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.551164 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.551148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa219405-3826-46fe-8c69-ea3042af755c-tls-certs\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.556168 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.556141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4xm\" (UniqueName: \"kubernetes.io/projected/fa219405-3826-46fe-8c69-ea3042af755c-kube-api-access-fz4xm\") pod \"premium-w4-56e305-kserve-78dc857c76-cth5z\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.658945 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.658859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:21.934443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:21.934369 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z"] Sep 15 21:43:21.936865 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:43:21.936831 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa219405_3826_46fe_8c69_ea3042af755c.slice/crio-d44a4a608d3ced1cc2f04b6b0bdf885def40e3daf29d2b2ff1d4f0a008d99c20 WatchSource:0}: Error finding container d44a4a608d3ced1cc2f04b6b0bdf885def40e3daf29d2b2ff1d4f0a008d99c20: Status 404 returned error can't find the container with id d44a4a608d3ced1cc2f04b6b0bdf885def40e3daf29d2b2ff1d4f0a008d99c20 Sep 15 21:43:22.938681 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:22.938646 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" event={"ID":"fa219405-3826-46fe-8c69-ea3042af755c","Type":"ContainerStarted","Data":"34dd85b52836297574b2d2a3537c9f899c6a3baf616b889740e51e82c049404a"} Sep 15 21:43:22.938681 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:22.938688 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" event={"ID":"fa219405-3826-46fe-8c69-ea3042af755c","Type":"ContainerStarted","Data":"d44a4a608d3ced1cc2f04b6b0bdf885def40e3daf29d2b2ff1d4f0a008d99c20"} Sep 15 21:43:22.954351 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:22.954289 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" podStartSLOduration=1.954267603 podStartE2EDuration="1.954267603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:22.953999827 +0000 UTC m=+1052.293479278" watchObservedRunningTime="2026-09-15 21:43:22.954267603 +0000 UTC m=+1052.293747056" Sep 15 21:43:31.659853 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:31.659809 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:31.660298 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:31.659879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:31.672702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:31.672674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:31.997283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:31.997254 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:43:59.561139 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.561015 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz"] Sep 15 21:43:59.566133 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.566093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.568966 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.568945 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2d4753a-serving-cert\"" Sep 15 21:43:59.572520 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.572490 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz"] Sep 15 21:43:59.712222 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.712180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnc5f\" (UniqueName: \"kubernetes.io/projected/6bfeb56e-9a64-4685-b172-5fde02a4ca56-kube-api-access-jnc5f\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.712425 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.712266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeb56e-9a64-4685-b172-5fde02a4ca56-openshift-service-ca\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.712425 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.712308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.712425 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.712333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-metrics-certs\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.813250 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.813156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeb56e-9a64-4685-b172-5fde02a4ca56-openshift-service-ca\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.813250 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.813203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.813250 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.813223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-metrics-certs\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.813561 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.813327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnc5f\" (UniqueName: \"kubernetes.io/projected/6bfeb56e-9a64-4685-b172-5fde02a4ca56-kube-api-access-jnc5f\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.813561 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:43:59.813339 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-serving-cert: secret "maas-api-e2e-mt-e2d4753a-serving-cert" not found Sep 15 21:43:59.813561 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:43:59.813412 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls podName:6bfeb56e-9a64-4685-b172-5fde02a4ca56 nodeName:}" failed. No retries permitted until 2026-09-15 21:44:00.313391329 +0000 UTC m=+1089.652870772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls") pod "maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" (UID: "6bfeb56e-9a64-4685-b172-5fde02a4ca56") : secret "maas-api-e2e-mt-e2d4753a-serving-cert" not found Sep 15 21:43:59.813897 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.813873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeb56e-9a64-4685-b172-5fde02a4ca56-openshift-service-ca\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.816308 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.816276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-metrics-certs\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:43:59.820964 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:43:59.820936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnc5f\" (UniqueName: \"kubernetes.io/projected/6bfeb56e-9a64-4685-b172-5fde02a4ca56-kube-api-access-jnc5f\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:00.318799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:00.318763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:00.321546 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:00.321512 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls\") pod \"maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:00.489717 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:00.489671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:00.771229 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:00.771205 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz"] Sep 15 21:44:00.773591 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:44:00.773546 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfeb56e_9a64_4685_b172_5fde02a4ca56.slice/crio-cfdb16e0108f754955dfbc2149dca70669f752bffc4e4b167420bcd3913304b1 WatchSource:0}: Error finding container cfdb16e0108f754955dfbc2149dca70669f752bffc4e4b167420bcd3913304b1: Status 404 returned error can't find the container with id cfdb16e0108f754955dfbc2149dca70669f752bffc4e4b167420bcd3913304b1 Sep 15 21:44:01.138855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:01.138823 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" event={"ID":"6bfeb56e-9a64-4685-b172-5fde02a4ca56","Type":"ContainerStarted","Data":"cfdb16e0108f754955dfbc2149dca70669f752bffc4e4b167420bcd3913304b1"} Sep 15 21:44:02.146046 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:02.146013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" event={"ID":"6bfeb56e-9a64-4685-b172-5fde02a4ca56","Type":"ContainerStarted","Data":"e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b"} Sep 15 21:44:02.146443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:02.146136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:02.163119 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:02.163055 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" podStartSLOduration=2.8381070040000003 podStartE2EDuration="3.163040282s" totalImagesPullingTime="324.933278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:02.161877593 +0000 UTC m=+1091.501357044" watchObservedRunningTime="2026-09-15 21:44:02.163040282 +0000 UTC m=+1091.502519729" Sep 15 21:44:08.156058 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:08.156023 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:38.216964 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.216927 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m"] Sep 15 21:44:38.217466 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.217220 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" podUID="a2f7745b-1a56-42f8-b62b-5965f97cf733" containerName="maas-api" containerID="cri-o://6ae0e78bd2c40562cd9e34a9d9222f25e1e5eba865d2ebc1b98649a3162f1844" gracePeriod=30 Sep 15 21:44:38.344181 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.344143 2575 generic.go:356] "Generic (PLEG): container finished" podID="a2f7745b-1a56-42f8-b62b-5965f97cf733" containerID="6ae0e78bd2c40562cd9e34a9d9222f25e1e5eba865d2ebc1b98649a3162f1844" exitCode=0 Sep 15 21:44:38.344326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.344195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" event={"ID":"a2f7745b-1a56-42f8-b62b-5965f97cf733","Type":"ContainerDied","Data":"6ae0e78bd2c40562cd9e34a9d9222f25e1e5eba865d2ebc1b98649a3162f1844"} Sep 15 21:44:38.491411 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.491384 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:44:38.595911 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.595879 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-metrics-certs\") pod \"a2f7745b-1a56-42f8-b62b-5965f97cf733\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " Sep 15 21:44:38.596139 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.595921 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dkrm\" (UniqueName: \"kubernetes.io/projected/a2f7745b-1a56-42f8-b62b-5965f97cf733-kube-api-access-5dkrm\") pod \"a2f7745b-1a56-42f8-b62b-5965f97cf733\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " Sep 15 21:44:38.596139 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.596076 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls\") pod \"a2f7745b-1a56-42f8-b62b-5965f97cf733\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " Sep 15 21:44:38.596139 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.596134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f7745b-1a56-42f8-b62b-5965f97cf733-openshift-service-ca\") pod \"a2f7745b-1a56-42f8-b62b-5965f97cf733\" (UID: \"a2f7745b-1a56-42f8-b62b-5965f97cf733\") " Sep 15 21:44:38.596630 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.596587 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f7745b-1a56-42f8-b62b-5965f97cf733-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2f7745b-1a56-42f8-b62b-5965f97cf733" (UID: "a2f7745b-1a56-42f8-b62b-5965f97cf733"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:44:38.598543 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.598502 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2f7745b-1a56-42f8-b62b-5965f97cf733" (UID: "a2f7745b-1a56-42f8-b62b-5965f97cf733"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:44:38.598748 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.598724 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2f7745b-1a56-42f8-b62b-5965f97cf733" (UID: "a2f7745b-1a56-42f8-b62b-5965f97cf733"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:44:38.598817 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.598807 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f7745b-1a56-42f8-b62b-5965f97cf733-kube-api-access-5dkrm" (OuterVolumeSpecName: "kube-api-access-5dkrm") pod "a2f7745b-1a56-42f8-b62b-5965f97cf733" (UID: "a2f7745b-1a56-42f8-b62b-5965f97cf733"). InnerVolumeSpecName "kube-api-access-5dkrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:38.697086 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.697054 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:38.697086 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.697082 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f7745b-1a56-42f8-b62b-5965f97cf733-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:38.697086 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.697091 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2f7745b-1a56-42f8-b62b-5965f97cf733-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:38.697349 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:38.697127 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dkrm\" (UniqueName: \"kubernetes.io/projected/a2f7745b-1a56-42f8-b62b-5965f97cf733-kube-api-access-5dkrm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:39.351282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:39.351242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" event={"ID":"a2f7745b-1a56-42f8-b62b-5965f97cf733","Type":"ContainerDied","Data":"cc245257e5c6673923b6f26e9d507c3f39dd955e724785ac958e6e6324e98ae8"} Sep 15 21:44:39.351282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:39.351276 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m" Sep 15 21:44:39.351877 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:39.351303 2575 scope.go:117] "RemoveContainer" containerID="6ae0e78bd2c40562cd9e34a9d9222f25e1e5eba865d2ebc1b98649a3162f1844" Sep 15 21:44:39.367375 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:39.367340 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m"] Sep 15 21:44:39.369297 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:39.369267 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a3a26-6f9d49cb46-v7k2m"] Sep 15 21:44:41.148016 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:41.147979 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f7745b-1a56-42f8-b62b-5965f97cf733" path="/var/lib/kubelet/pods/a2f7745b-1a56-42f8-b62b-5965f97cf733/volumes" Sep 15 21:44:44.077117 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.077047 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz"] Sep 15 21:44:44.077848 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.077814 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" podUID="6bfeb56e-9a64-4685-b172-5fde02a4ca56" containerName="maas-api" containerID="cri-o://e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b" gracePeriod=30 Sep 15 21:44:44.337729 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.337654 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:44.381247 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.381212 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bfeb56e-9a64-4685-b172-5fde02a4ca56" containerID="e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b" exitCode=0 Sep 15 21:44:44.381476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.381286 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" Sep 15 21:44:44.381476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.381297 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" event={"ID":"6bfeb56e-9a64-4685-b172-5fde02a4ca56","Type":"ContainerDied","Data":"e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b"} Sep 15 21:44:44.381476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.381375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz" event={"ID":"6bfeb56e-9a64-4685-b172-5fde02a4ca56","Type":"ContainerDied","Data":"cfdb16e0108f754955dfbc2149dca70669f752bffc4e4b167420bcd3913304b1"} Sep 15 21:44:44.381476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.381427 2575 scope.go:117] "RemoveContainer" containerID="e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b" Sep 15 21:44:44.391476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.391457 2575 scope.go:117] "RemoveContainer" containerID="e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b" Sep 15 21:44:44.391830 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:44:44.391806 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b\": container with ID starting with e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b not found: ID does not exist" containerID="e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b" Sep 15 21:44:44.391885 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.391846 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b"} err="failed to get container status \"e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b\": rpc error: code = NotFound desc = could not find container \"e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b\": container with ID starting with e0d94b46887fa0cd07257307a059152c89cde775e998dc345daef2b51677035b not found: ID does not exist" Sep 15 21:44:44.453354 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.453325 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls\") pod \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " Sep 15 21:44:44.453543 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.453398 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-metrics-certs\") pod \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " Sep 15 21:44:44.453543 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.453499 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnc5f\" (UniqueName: \"kubernetes.io/projected/6bfeb56e-9a64-4685-b172-5fde02a4ca56-kube-api-access-jnc5f\") pod \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " Sep 15 21:44:44.453543 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.453538 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeb56e-9a64-4685-b172-5fde02a4ca56-openshift-service-ca\") pod \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\" (UID: \"6bfeb56e-9a64-4685-b172-5fde02a4ca56\") " Sep 15 21:44:44.454035 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.454007 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfeb56e-9a64-4685-b172-5fde02a4ca56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bfeb56e-9a64-4685-b172-5fde02a4ca56" (UID: "6bfeb56e-9a64-4685-b172-5fde02a4ca56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:44:44.455734 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.455711 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfeb56e-9a64-4685-b172-5fde02a4ca56-kube-api-access-jnc5f" (OuterVolumeSpecName: "kube-api-access-jnc5f") pod "6bfeb56e-9a64-4685-b172-5fde02a4ca56" (UID: "6bfeb56e-9a64-4685-b172-5fde02a4ca56"). InnerVolumeSpecName "kube-api-access-jnc5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:44.456146 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.456120 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6bfeb56e-9a64-4685-b172-5fde02a4ca56" (UID: "6bfeb56e-9a64-4685-b172-5fde02a4ca56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:44:44.456146 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.456142 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bfeb56e-9a64-4685-b172-5fde02a4ca56" (UID: "6bfeb56e-9a64-4685-b172-5fde02a4ca56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:44:44.554272 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.554232 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:44.554272 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.554266 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnc5f\" (UniqueName: \"kubernetes.io/projected/6bfeb56e-9a64-4685-b172-5fde02a4ca56-kube-api-access-jnc5f\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:44.554272 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.554279 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeb56e-9a64-4685-b172-5fde02a4ca56-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:44.554516 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.554291 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfeb56e-9a64-4685-b172-5fde02a4ca56-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:44:44.705118 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.705075 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz"] Sep 15 21:44:44.711211 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:44.711183 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2d4753a-dd994bc79-dtjbz"] Sep 15 21:44:45.147806 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:44:45.147726 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bfeb56e-9a64-4685-b172-5fde02a4ca56" path="/var/lib/kubelet/pods/6bfeb56e-9a64-4685-b172-5fde02a4ca56/volumes" Sep 15 21:45:10.833857 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.833817 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59"] Sep 15 21:45:10.834457 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.834434 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfeb56e-9a64-4685-b172-5fde02a4ca56" containerName="maas-api" Sep 15 21:45:10.834457 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.834454 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfeb56e-9a64-4685-b172-5fde02a4ca56" containerName="maas-api" Sep 15 21:45:10.834578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.834485 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f7745b-1a56-42f8-b62b-5965f97cf733" containerName="maas-api" Sep 15 21:45:10.834578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.834494 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f7745b-1a56-42f8-b62b-5965f97cf733" containerName="maas-api" Sep 15 21:45:10.834674 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.834608 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f7745b-1a56-42f8-b62b-5965f97cf733" containerName="maas-api" Sep 15 21:45:10.834674 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.834623 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfeb56e-9a64-4685-b172-5fde02a4ca56" containerName="maas-api" Sep 15 21:45:10.839399 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.839377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:10.841921 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.841899 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-850f22-serving-cert\"" Sep 15 21:45:10.847454 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.847431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59"] Sep 15 21:45:10.905965 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.905935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:10.906176 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.905980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c09de5-9e48-4b1d-8810-de528c541431-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:10.906176 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.906000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smsr\" (UniqueName: \"kubernetes.io/projected/26c09de5-9e48-4b1d-8810-de528c541431-kube-api-access-7smsr\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:10.906176 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:10.906029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.007448 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.007412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.007652 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.007461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c09de5-9e48-4b1d-8810-de528c541431-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.007652 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.007480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smsr\" (UniqueName: \"kubernetes.io/projected/26c09de5-9e48-4b1d-8810-de528c541431-kube-api-access-7smsr\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.007652 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.007501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.007652 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:45:11.007588 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-serving-cert: secret "maas-api-e2e-shared-a-w2-850f22-serving-cert" not found Sep 15 21:45:11.007876 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:45:11.007674 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls podName:26c09de5-9e48-4b1d-8810-de528c541431 nodeName:}" failed. No retries permitted until 2026-09-15 21:45:11.507655267 +0000 UTC m=+1160.847134696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls") pod "maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" (UID: "26c09de5-9e48-4b1d-8810-de528c541431") : secret "maas-api-e2e-shared-a-w2-850f22-serving-cert" not found Sep 15 21:45:11.008238 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.008215 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c09de5-9e48-4b1d-8810-de528c541431-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.010451 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.010422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.016782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.016758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smsr\" (UniqueName: \"kubernetes.io/projected/26c09de5-9e48-4b1d-8810-de528c541431-kube-api-access-7smsr\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.513443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:11.513402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:11.513611 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:45:11.513556 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-serving-cert: secret "maas-api-e2e-shared-a-w2-850f22-serving-cert" not found Sep 15 21:45:11.513660 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:45:11.513623 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls podName:26c09de5-9e48-4b1d-8810-de528c541431 nodeName:}" failed. No retries permitted until 2026-09-15 21:45:12.51360795 +0000 UTC m=+1161.853087379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls") pod "maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" (UID: "26c09de5-9e48-4b1d-8810-de528c541431") : secret "maas-api-e2e-shared-a-w2-850f22-serving-cert" not found Sep 15 21:45:12.523789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:12.523753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:12.526469 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:12.526443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:12.655394 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:12.655354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:12.929855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:12.929828 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59"] Sep 15 21:45:12.931168 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:45:12.931136 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c09de5_9e48_4b1d_8810_de528c541431.slice/crio-fb2caa2930f18c10d01f16ca81dc776a03124503f0f026351ae5f2fb95c30c9e WatchSource:0}: Error finding container fb2caa2930f18c10d01f16ca81dc776a03124503f0f026351ae5f2fb95c30c9e: Status 404 returned error can't find the container with id fb2caa2930f18c10d01f16ca81dc776a03124503f0f026351ae5f2fb95c30c9e Sep 15 21:45:13.538629 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:13.538598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" event={"ID":"26c09de5-9e48-4b1d-8810-de528c541431","Type":"ContainerStarted","Data":"fb2caa2930f18c10d01f16ca81dc776a03124503f0f026351ae5f2fb95c30c9e"} Sep 15 21:45:14.544866 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:14.544828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" event={"ID":"26c09de5-9e48-4b1d-8810-de528c541431","Type":"ContainerStarted","Data":"4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c"} Sep 15 21:45:14.545311 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:14.545030 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:14.562991 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:14.562933 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" podStartSLOduration=3.991820105 podStartE2EDuration="4.562912634s" totalImagesPullingTime="571.092529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:14.562032975 +0000 UTC m=+1163.901512427" watchObservedRunningTime="2026-09-15 21:45:14.562912634 +0000 UTC m=+1163.902392088" Sep 15 21:45:20.555347 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:20.555307 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:45:45.228679 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.228639 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh"] Sep 15 21:45:45.233141 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.233093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.235557 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.235533 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-8c1d2a-serving-cert\"" Sep 15 21:45:45.238516 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.238495 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh"] Sep 15 21:45:45.348240 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.348207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.348414 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.348257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.348414 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.348284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzjw\" (UniqueName: \"kubernetes.io/projected/daa2c63c-388e-4045-bca3-e904dc2b90ff-kube-api-access-tbzjw\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.348414 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.348345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa2c63c-388e-4045-bca3-e904dc2b90ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.448860 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.448821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.448860 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.448870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.449129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.448902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzjw\" (UniqueName: \"kubernetes.io/projected/daa2c63c-388e-4045-bca3-e904dc2b90ff-kube-api-access-tbzjw\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.449129 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.448921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa2c63c-388e-4045-bca3-e904dc2b90ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.449605 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.449585 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa2c63c-388e-4045-bca3-e904dc2b90ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.451427 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.451399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.451427 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.451422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.456956 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.456933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzjw\" (UniqueName: \"kubernetes.io/projected/daa2c63c-388e-4045-bca3-e904dc2b90ff-kube-api-access-tbzjw\") pod \"maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.546305 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.546223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:45.823076 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:45.823042 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh"] Sep 15 21:45:45.824545 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:45:45.824521 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa2c63c_388e_4045_bca3_e904dc2b90ff.slice/crio-bbbf8f24628893e2c90f62eb51a84653cdad232fa3700c31245fef76dbf30b3b WatchSource:0}: Error finding container bbbf8f24628893e2c90f62eb51a84653cdad232fa3700c31245fef76dbf30b3b: Status 404 returned error can't find the container with id bbbf8f24628893e2c90f62eb51a84653cdad232fa3700c31245fef76dbf30b3b Sep 15 21:45:46.716010 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:46.715978 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" event={"ID":"daa2c63c-388e-4045-bca3-e904dc2b90ff","Type":"ContainerStarted","Data":"377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c"} Sep 15 21:45:46.716427 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:46.716029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" event={"ID":"daa2c63c-388e-4045-bca3-e904dc2b90ff","Type":"ContainerStarted","Data":"bbbf8f24628893e2c90f62eb51a84653cdad232fa3700c31245fef76dbf30b3b"} Sep 15 21:45:46.716427 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:46.716066 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:45:46.734455 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:46.734406 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" podStartSLOduration=1.372473791 podStartE2EDuration="1.734390132s" totalImagesPullingTime="361.916341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:46.73235179 +0000 UTC m=+1196.071831242" watchObservedRunningTime="2026-09-15 21:45:46.734390132 +0000 UTC m=+1196.073869614" Sep 15 21:45:52.726333 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:45:52.726304 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:46:11.926538 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:11.926504 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp"] Sep 15 21:46:11.930402 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:11.930386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:11.933843 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:11.933819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"distinct-w0-360e43-kserve-self-signed-certs\"" Sep 15 21:46:11.940586 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:11.940564 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp"] Sep 15 21:46:12.009058 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.009026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7eede26-79bf-432b-b2fb-57973c63749f-tls-certs\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.009282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.009085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-tmp-dir\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.009282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.009182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-model-cache\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.009405 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.009293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgwq\" (UniqueName: \"kubernetes.io/projected/d7eede26-79bf-432b-b2fb-57973c63749f-kube-api-access-tsgwq\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.009405 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.009328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-dshm\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.009405 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.009388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-home\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgwq\" (UniqueName: \"kubernetes.io/projected/d7eede26-79bf-432b-b2fb-57973c63749f-kube-api-access-tsgwq\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110519 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-dshm\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110589 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-home\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7eede26-79bf-432b-b2fb-57973c63749f-tls-certs\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110724 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-tmp-dir\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-model-cache\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.110930 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.110906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-home\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.111123 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.111041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-model-cache\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.111211 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.111083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-tmp-dir\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.113400 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.113378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-dshm\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.113512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.113440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7eede26-79bf-432b-b2fb-57973c63749f-tls-certs\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.118392 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.118373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgwq\" (UniqueName: \"kubernetes.io/projected/d7eede26-79bf-432b-b2fb-57973c63749f-kube-api-access-tsgwq\") pod \"distinct-w0-360e43-kserve-5f5f769fd8-hkphp\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.240338 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.240256 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:12.518724 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.518692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp"] Sep 15 21:46:12.520740 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:46:12.520709 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7eede26_79bf_432b_b2fb_57973c63749f.slice/crio-c4fb65ba932549b3fcb091c3c34a6c0e72818b99832c5e26d36059b40f075a38 WatchSource:0}: Error finding container c4fb65ba932549b3fcb091c3c34a6c0e72818b99832c5e26d36059b40f075a38: Status 404 returned error can't find the container with id c4fb65ba932549b3fcb091c3c34a6c0e72818b99832c5e26d36059b40f075a38 Sep 15 21:46:12.850468 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.850369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" event={"ID":"d7eede26-79bf-432b-b2fb-57973c63749f","Type":"ContainerStarted","Data":"9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab"} Sep 15 21:46:12.850468 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.850420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" event={"ID":"d7eede26-79bf-432b-b2fb-57973c63749f","Type":"ContainerStarted","Data":"c4fb65ba932549b3fcb091c3c34a6c0e72818b99832c5e26d36059b40f075a38"} Sep 15 21:46:12.872360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:12.872310 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" podStartSLOduration=1.872294718 podStartE2EDuration="1.872294718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:12.87024006 +0000 UTC m=+1222.209719511" watchObservedRunningTime="2026-09-15 21:46:12.872294718 +0000 UTC m=+1222.211774169" Sep 15 21:46:22.241046 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:22.241005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:22.241549 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:22.241077 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:22.253464 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:22.253440 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:22.915939 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:22.915905 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:46:28.644640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.644606 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp"] Sep 15 21:46:28.648607 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.648587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.651080 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.651056 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-459b305b-serving-cert\"" Sep 15 21:46:28.657730 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.657709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp"] Sep 15 21:46:28.771154 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.771116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-openshift-service-ca\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.771342 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.771167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnt8x\" (UniqueName: \"kubernetes.io/projected/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-kube-api-access-xnt8x\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.771342 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.771200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-metrics-certs\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.771342 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.771246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.872267 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.872237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-openshift-service-ca\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.872267 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.872270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnt8x\" (UniqueName: \"kubernetes.io/projected/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-kube-api-access-xnt8x\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.872552 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.872292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-metrics-certs\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.872552 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.872323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.872552 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:46:28.872463 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-serving-cert: secret "maas-api-e2e-mt-459b305b-serving-cert" not found Sep 15 21:46:28.872552 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:46:28.872522 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls podName:87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e nodeName:}" failed. No retries permitted until 2026-09-15 21:46:29.372505753 +0000 UTC m=+1238.711985201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls") pod "maas-api-e2e-mt-459b305b-856bc55878-g28dp" (UID: "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e") : secret "maas-api-e2e-mt-459b305b-serving-cert" not found Sep 15 21:46:28.873004 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.872976 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-openshift-service-ca\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.874860 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.874837 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-metrics-certs\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:28.881788 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:28.881760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnt8x\" (UniqueName: \"kubernetes.io/projected/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-kube-api-access-xnt8x\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:29.378262 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:29.378222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:29.381128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:29.381074 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls\") pod \"maas-api-e2e-mt-459b305b-856bc55878-g28dp\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:29.560262 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:29.560227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:29.840685 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:29.840647 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp"] Sep 15 21:46:29.842496 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:46:29.842470 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fc2ada_77ce_4a3f_b13c_aa6b2e2b4d5e.slice/crio-e3b8bc5f1668e48d58d7ffc6a5a62482eed6ad32dd3dca0ee1dfe6a4e4f7343c WatchSource:0}: Error finding container e3b8bc5f1668e48d58d7ffc6a5a62482eed6ad32dd3dca0ee1dfe6a4e4f7343c: Status 404 returned error can't find the container with id e3b8bc5f1668e48d58d7ffc6a5a62482eed6ad32dd3dca0ee1dfe6a4e4f7343c Sep 15 21:46:29.941814 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:29.941780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" event={"ID":"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e","Type":"ContainerStarted","Data":"e3b8bc5f1668e48d58d7ffc6a5a62482eed6ad32dd3dca0ee1dfe6a4e4f7343c"} Sep 15 21:46:30.949496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:30.949460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" event={"ID":"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e","Type":"ContainerStarted","Data":"7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848"} Sep 15 21:46:30.949897 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:30.949603 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:30.968474 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:30.968425 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" podStartSLOduration=2.546340651 podStartE2EDuration="2.968411036s" totalImagesPullingTime="422.070385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:30.966909879 +0000 UTC m=+1240.306389329" watchObservedRunningTime="2026-09-15 21:46:30.968411036 +0000 UTC m=+1240.307890516" Sep 15 21:46:36.961373 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:36.961344 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:46:50.219894 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.219802 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6"] Sep 15 21:46:50.223247 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.223224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.225848 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.225824 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-87ba8b\"/\"kube-root-ca.crt\"" Sep 15 21:46:50.226782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.226761 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-87ba8b\"/\"openshift-service-ca.crt\"" Sep 15 21:46:50.226910 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.226815 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-87ba8b\"/\"noten-model-87ba8b-kserve-self-signed-certs\"" Sep 15 21:46:50.226975 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.226949 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-87ba8b\"/\"default-dockercfg-8rt6h\"" Sep 15 21:46:50.233789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.233764 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6"] Sep 15 21:46:50.393191 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.393156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22lhw\" (UniqueName: \"kubernetes.io/projected/bcc7081d-428b-47d5-87fd-06b76614bbda-kube-api-access-22lhw\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.393384 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.393304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-home\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.393384 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.393355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc7081d-428b-47d5-87fd-06b76614bbda-tls-certs\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.393490 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.393388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-model-cache\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.393490 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.393433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-tmp-dir\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.393595 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.393545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-dshm\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.494959 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.494863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-home\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.494959 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.494907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc7081d-428b-47d5-87fd-06b76614bbda-tls-certs\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495237 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-model-cache\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495237 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-tmp-dir\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495356 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-dshm\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495356 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22lhw\" (UniqueName: \"kubernetes.io/projected/bcc7081d-428b-47d5-87fd-06b76614bbda-kube-api-access-22lhw\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495583 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-home\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495754 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-tmp-dir\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.495754 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.495628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-model-cache\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.498147 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.498088 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-dshm\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.498311 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.498292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc7081d-428b-47d5-87fd-06b76614bbda-tls-certs\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.503468 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.503446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22lhw\" (UniqueName: \"kubernetes.io/projected/bcc7081d-428b-47d5-87fd-06b76614bbda-kube-api-access-22lhw\") pod \"noten-model-87ba8b-kserve-74777c8655-s6hw6\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.534495 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.534463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:46:50.810945 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:50.810916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6"] Sep 15 21:46:50.812853 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:46:50.812830 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcc7081d_428b_47d5_87fd_06b76614bbda.slice/crio-40486abf03c5d7cd972e0af64e87c38a4e6a75de6829273b6a3c9760d73873aa WatchSource:0}: Error finding container 40486abf03c5d7cd972e0af64e87c38a4e6a75de6829273b6a3c9760d73873aa: Status 404 returned error can't find the container with id 40486abf03c5d7cd972e0af64e87c38a4e6a75de6829273b6a3c9760d73873aa Sep 15 21:46:51.061940 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:51.061865 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" event={"ID":"bcc7081d-428b-47d5-87fd-06b76614bbda","Type":"ContainerStarted","Data":"d3b554d809caa5c516494f35545bde9485298900f70f5c028a8a142c98bee2a2"} Sep 15 21:46:51.061940 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:51.061908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" event={"ID":"bcc7081d-428b-47d5-87fd-06b76614bbda","Type":"ContainerStarted","Data":"40486abf03c5d7cd972e0af64e87c38a4e6a75de6829273b6a3c9760d73873aa"} Sep 15 21:46:51.079978 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:46:51.079920 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" podStartSLOduration=1.079896234 podStartE2EDuration="1.079896234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:51.077268957 +0000 UTC m=+1260.416748408" watchObservedRunningTime="2026-09-15 21:46:51.079896234 +0000 UTC m=+1260.419375685" Sep 15 21:47:00.534749 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:00.534710 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:47:00.535291 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:00.534767 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:47:00.547673 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:00.547646 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:47:01.136727 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:01.136698 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:47:19.082937 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:19.082885 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6"] Sep 15 21:47:31.053816 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.053783 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5"] Sep 15 21:47:31.054295 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.054010 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" podUID="a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" containerName="maas-api" containerID="cri-o://67bb6f928906bf412e48195418ea87b0dbff1b144b35e8ce6661f15c8d617453" gracePeriod=30 Sep 15 21:47:31.285283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.285247 2575 generic.go:356] "Generic (PLEG): container finished" podID="a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" containerID="67bb6f928906bf412e48195418ea87b0dbff1b144b35e8ce6661f15c8d617453" exitCode=0 Sep 15 21:47:31.285474 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.285309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" event={"ID":"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec","Type":"ContainerDied","Data":"67bb6f928906bf412e48195418ea87b0dbff1b144b35e8ce6661f15c8d617453"} Sep 15 21:47:31.321151 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.321122 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:47:31.507436 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.507400 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-openshift-service-ca\") pod \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " Sep 15 21:47:31.507623 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.507517 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-metrics-certs\") pod \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " Sep 15 21:47:31.507623 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.507573 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls\") pod \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " Sep 15 21:47:31.507761 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.507639 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcn7t\" (UniqueName: \"kubernetes.io/projected/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-kube-api-access-mcn7t\") pod \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\" (UID: \"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec\") " Sep 15 21:47:31.507761 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.507674 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" (UID: "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:47:31.507987 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.507969 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:31.509786 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.509755 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-kube-api-access-mcn7t" (OuterVolumeSpecName: "kube-api-access-mcn7t") pod "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" (UID: "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec"). InnerVolumeSpecName "kube-api-access-mcn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:47:31.509786 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.509781 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" (UID: "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:47:31.509926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.509840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" (UID: "a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:47:31.609417 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.609339 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:31.609417 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.609368 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:31.609417 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:31.609377 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcn7t\" (UniqueName: \"kubernetes.io/projected/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec-kube-api-access-mcn7t\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:32.293745 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:32.293712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" event={"ID":"a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec","Type":"ContainerDied","Data":"123b44d5fc3623bee722656a957c20496ddeb364842ed1a94880562596ef3cb4"} Sep 15 21:47:32.294313 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:32.293753 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5" Sep 15 21:47:32.294313 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:32.293787 2575 scope.go:117] "RemoveContainer" containerID="67bb6f928906bf412e48195418ea87b0dbff1b144b35e8ce6661f15c8d617453" Sep 15 21:47:32.330710 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:32.330682 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5"] Sep 15 21:47:32.338874 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:32.338846 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1af1db-59b8f596df-x72n5"] Sep 15 21:47:33.148135 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:33.148081 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" path="/var/lib/kubelet/pods/a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec/volumes" Sep 15 21:47:34.105980 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.105931 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" podUID="bcc7081d-428b-47d5-87fd-06b76614bbda" containerName="main" containerID="cri-o://d3b554d809caa5c516494f35545bde9485298900f70f5c028a8a142c98bee2a2" gracePeriod=45 Sep 15 21:47:34.310170 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.310128 2575 generic.go:356] "Generic (PLEG): container finished" podID="bcc7081d-428b-47d5-87fd-06b76614bbda" containerID="d3b554d809caa5c516494f35545bde9485298900f70f5c028a8a142c98bee2a2" exitCode=0 Sep 15 21:47:34.310319 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.310148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" event={"ID":"bcc7081d-428b-47d5-87fd-06b76614bbda","Type":"ContainerDied","Data":"d3b554d809caa5c516494f35545bde9485298900f70f5c028a8a142c98bee2a2"} Sep 15 21:47:34.371292 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.371231 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:47:34.437874 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.437845 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22lhw\" (UniqueName: \"kubernetes.io/projected/bcc7081d-428b-47d5-87fd-06b76614bbda-kube-api-access-22lhw\") pod \"bcc7081d-428b-47d5-87fd-06b76614bbda\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " Sep 15 21:47:34.438044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.437882 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-model-cache\") pod \"bcc7081d-428b-47d5-87fd-06b76614bbda\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " Sep 15 21:47:34.438044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.437924 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-tmp-dir\") pod \"bcc7081d-428b-47d5-87fd-06b76614bbda\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " Sep 15 21:47:34.438044 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.437970 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-home\") pod \"bcc7081d-428b-47d5-87fd-06b76614bbda\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " Sep 15 21:47:34.438239 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc7081d-428b-47d5-87fd-06b76614bbda-tls-certs\") pod \"bcc7081d-428b-47d5-87fd-06b76614bbda\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " Sep 15 21:47:34.438239 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438089 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-dshm\") pod \"bcc7081d-428b-47d5-87fd-06b76614bbda\" (UID: \"bcc7081d-428b-47d5-87fd-06b76614bbda\") " Sep 15 21:47:34.438239 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438193 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcc7081d-428b-47d5-87fd-06b76614bbda" (UID: "bcc7081d-428b-47d5-87fd-06b76614bbda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:47:34.438402 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438225 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcc7081d-428b-47d5-87fd-06b76614bbda" (UID: "bcc7081d-428b-47d5-87fd-06b76614bbda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:47:34.438504 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438485 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:34.438570 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438509 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:34.438664 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.438642 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-home" (OuterVolumeSpecName: "home") pod "bcc7081d-428b-47d5-87fd-06b76614bbda" (UID: "bcc7081d-428b-47d5-87fd-06b76614bbda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:47:34.440392 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.440360 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc7081d-428b-47d5-87fd-06b76614bbda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcc7081d-428b-47d5-87fd-06b76614bbda" (UID: "bcc7081d-428b-47d5-87fd-06b76614bbda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:47:34.440778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.440756 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-dshm" (OuterVolumeSpecName: "dshm") pod "bcc7081d-428b-47d5-87fd-06b76614bbda" (UID: "bcc7081d-428b-47d5-87fd-06b76614bbda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:47:34.440778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.440757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc7081d-428b-47d5-87fd-06b76614bbda-kube-api-access-22lhw" (OuterVolumeSpecName: "kube-api-access-22lhw") pod "bcc7081d-428b-47d5-87fd-06b76614bbda" (UID: "bcc7081d-428b-47d5-87fd-06b76614bbda"). InnerVolumeSpecName "kube-api-access-22lhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:47:34.539944 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.539910 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:34.539944 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.539942 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22lhw\" (UniqueName: \"kubernetes.io/projected/bcc7081d-428b-47d5-87fd-06b76614bbda-kube-api-access-22lhw\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:34.539944 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.539953 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcc7081d-428b-47d5-87fd-06b76614bbda-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:34.540199 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:34.539961 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc7081d-428b-47d5-87fd-06b76614bbda-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:35.317588 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:35.317556 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" event={"ID":"bcc7081d-428b-47d5-87fd-06b76614bbda","Type":"ContainerDied","Data":"40486abf03c5d7cd972e0af64e87c38a4e6a75de6829273b6a3c9760d73873aa"} Sep 15 21:47:35.317588 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:35.317591 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6" Sep 15 21:47:35.318078 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:35.317601 2575 scope.go:117] "RemoveContainer" containerID="d3b554d809caa5c516494f35545bde9485298900f70f5c028a8a142c98bee2a2" Sep 15 21:47:35.333183 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:35.333159 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6"] Sep 15 21:47:35.335064 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:35.335044 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-87ba8b/noten-model-87ba8b-kserve-74777c8655-s6hw6"] Sep 15 21:47:37.148225 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:37.148188 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc7081d-428b-47d5-87fd-06b76614bbda" path="/var/lib/kubelet/pods/bcc7081d-428b-47d5-87fd-06b76614bbda/volumes" Sep 15 21:47:48.349790 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.349757 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w"] Sep 15 21:47:48.350308 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.350021 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" podUID="b0ae0d9c-12a4-4822-a256-78d731730512" containerName="maas-api" containerID="cri-o://3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4" gracePeriod=30 Sep 15 21:47:48.601757 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.601701 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:47:48.681548 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.681519 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-metrics-certs\") pod \"b0ae0d9c-12a4-4822-a256-78d731730512\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " Sep 15 21:47:48.681725 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.681590 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0d9c-12a4-4822-a256-78d731730512-openshift-service-ca\") pod \"b0ae0d9c-12a4-4822-a256-78d731730512\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " Sep 15 21:47:48.681725 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.681648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwlph\" (UniqueName: \"kubernetes.io/projected/b0ae0d9c-12a4-4822-a256-78d731730512-kube-api-access-vwlph\") pod \"b0ae0d9c-12a4-4822-a256-78d731730512\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " Sep 15 21:47:48.681725 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.681681 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-maas-api-tls\") pod \"b0ae0d9c-12a4-4822-a256-78d731730512\" (UID: \"b0ae0d9c-12a4-4822-a256-78d731730512\") " Sep 15 21:47:48.682068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.681968 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ae0d9c-12a4-4822-a256-78d731730512-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0ae0d9c-12a4-4822-a256-78d731730512" (UID: "b0ae0d9c-12a4-4822-a256-78d731730512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:47:48.682068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.682040 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0d9c-12a4-4822-a256-78d731730512-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:48.684068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.684042 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0ae0d9c-12a4-4822-a256-78d731730512" (UID: "b0ae0d9c-12a4-4822-a256-78d731730512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:47:48.684068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.684057 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ae0d9c-12a4-4822-a256-78d731730512-kube-api-access-vwlph" (OuterVolumeSpecName: "kube-api-access-vwlph") pod "b0ae0d9c-12a4-4822-a256-78d731730512" (UID: "b0ae0d9c-12a4-4822-a256-78d731730512"). InnerVolumeSpecName "kube-api-access-vwlph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:47:48.684261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.684203 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0ae0d9c-12a4-4822-a256-78d731730512" (UID: "b0ae0d9c-12a4-4822-a256-78d731730512"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:47:48.783063 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.783019 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:48.783063 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.783048 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlph\" (UniqueName: \"kubernetes.io/projected/b0ae0d9c-12a4-4822-a256-78d731730512-kube-api-access-vwlph\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:48.783063 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:48.783061 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0d9c-12a4-4822-a256-78d731730512-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:47:49.394159 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.394092 2575 generic.go:356] "Generic (PLEG): container finished" podID="b0ae0d9c-12a4-4822-a256-78d731730512" containerID="3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4" exitCode=0 Sep 15 21:47:49.394159 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.394141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" event={"ID":"b0ae0d9c-12a4-4822-a256-78d731730512","Type":"ContainerDied","Data":"3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4"} Sep 15 21:47:49.394159 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.394165 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" Sep 15 21:47:49.394741 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.394220 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w" event={"ID":"b0ae0d9c-12a4-4822-a256-78d731730512","Type":"ContainerDied","Data":"aaf68a82446c052c5da0f8dabf0a3f6fbac3e97a031c5d07a514f81c79f026a6"} Sep 15 21:47:49.394741 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.394256 2575 scope.go:117] "RemoveContainer" containerID="3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4" Sep 15 21:47:49.403046 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.403029 2575 scope.go:117] "RemoveContainer" containerID="3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4" Sep 15 21:47:49.403397 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:47:49.403376 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4\": container with ID starting with 3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4 not found: ID does not exist" containerID="3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4" Sep 15 21:47:49.403475 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.403405 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4"} err="failed to get container status \"3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4\": rpc error: code = NotFound desc = could not find container \"3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4\": container with ID starting with 3680c8a1f817596959b29bfa1c900b6516a47c408997c228962990983a43eca4 not found: ID does not exist" Sep 15 21:47:49.417425 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.417396 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w"] Sep 15 21:47:49.423282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:49.423259 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2d9c4-5c88d557f8-h7j6w"] Sep 15 21:47:51.148077 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:47:51.148045 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ae0d9c-12a4-4822-a256-78d731730512" path="/var/lib/kubelet/pods/b0ae0d9c-12a4-4822-a256-78d731730512/volumes" Sep 15 21:48:59.632749 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.632712 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2"] Sep 15 21:48:59.633366 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633347 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc7081d-428b-47d5-87fd-06b76614bbda" containerName="main" Sep 15 21:48:59.633439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633369 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc7081d-428b-47d5-87fd-06b76614bbda" containerName="main" Sep 15 21:48:59.633439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633395 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ae0d9c-12a4-4822-a256-78d731730512" containerName="maas-api" Sep 15 21:48:59.633439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633406 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ae0d9c-12a4-4822-a256-78d731730512" containerName="maas-api" Sep 15 21:48:59.633439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633417 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" containerName="maas-api" Sep 15 21:48:59.633439 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633425 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" containerName="maas-api" Sep 15 21:48:59.633705 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633517 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc7081d-428b-47d5-87fd-06b76614bbda" containerName="main" Sep 15 21:48:59.633705 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633528 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ae0d9c-12a4-4822-a256-78d731730512" containerName="maas-api" Sep 15 21:48:59.633705 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.633539 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f11aa0-2a2f-4ab7-a41e-4f52fddab9ec" containerName="maas-api" Sep 15 21:48:59.637390 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.637321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.640007 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.639977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"distinct-2-w0-360e43-kserve-self-signed-certs\"" Sep 15 21:48:59.651261 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.651167 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2"] Sep 15 21:48:59.655164 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.655066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72862e-2fa9-463b-9d1b-c694e803c03e-tls-certs\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.655584 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.655556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-dshm\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.655805 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.655782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlvl\" (UniqueName: \"kubernetes.io/projected/3c72862e-2fa9-463b-9d1b-c694e803c03e-kube-api-access-pzlvl\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.655997 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.655979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-home\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.656190 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.656169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-tmp-dir\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.656386 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.656368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-model-cache\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.756946 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.756914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-tmp-dir\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757147 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.756976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-model-cache\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757147 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72862e-2fa9-463b-9d1b-c694e803c03e-tls-certs\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757147 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-dshm\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757147 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlvl\" (UniqueName: \"kubernetes.io/projected/3c72862e-2fa9-463b-9d1b-c694e803c03e-kube-api-access-pzlvl\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757147 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-home\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757421 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-tmp-dir\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-model-cache\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.757605 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.757582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-home\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.759613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.759590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-dshm\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.759859 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.759838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72862e-2fa9-463b-9d1b-c694e803c03e-tls-certs\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.764432 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.764408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlvl\" (UniqueName: \"kubernetes.io/projected/3c72862e-2fa9-463b-9d1b-c694e803c03e-kube-api-access-pzlvl\") pod \"distinct-2-w0-360e43-kserve-ff965fc99-k47n2\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:48:59.955171 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:48:59.955141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:49:00.228939 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:00.228854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2"] Sep 15 21:49:00.232294 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:49:00.232264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c72862e_2fa9_463b_9d1b_c694e803c03e.slice/crio-395090591b87395df82854d32e2838330948be41eddc7a9a5ed717c7bd9e31b7 WatchSource:0}: Error finding container 395090591b87395df82854d32e2838330948be41eddc7a9a5ed717c7bd9e31b7: Status 404 returned error can't find the container with id 395090591b87395df82854d32e2838330948be41eddc7a9a5ed717c7bd9e31b7 Sep 15 21:49:00.766137 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:00.766084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" event={"ID":"3c72862e-2fa9-463b-9d1b-c694e803c03e","Type":"ContainerStarted","Data":"44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c"} Sep 15 21:49:00.766535 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:00.766154 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" event={"ID":"3c72862e-2fa9-463b-9d1b-c694e803c03e","Type":"ContainerStarted","Data":"395090591b87395df82854d32e2838330948be41eddc7a9a5ed717c7bd9e31b7"} Sep 15 21:49:00.781917 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:00.781872 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" podStartSLOduration=1.781856451 podStartE2EDuration="1.781856451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:49:00.781365044 +0000 UTC m=+1390.120844494" watchObservedRunningTime="2026-09-15 21:49:00.781856451 +0000 UTC m=+1390.121335901" Sep 15 21:49:09.956111 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:09.956073 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:49:09.956597 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:09.956148 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:49:09.968703 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:09.968681 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:49:10.832041 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:10.832002 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:49:52.158999 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.158903 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp"] Sep 15 21:49:52.159572 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.159171 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" podUID="87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" containerName="maas-api" containerID="cri-o://7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848" gracePeriod=30 Sep 15 21:49:52.407319 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.407295 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:49:52.466561 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.466530 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-openshift-service-ca\") pod \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " Sep 15 21:49:52.466736 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.466639 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnt8x\" (UniqueName: \"kubernetes.io/projected/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-kube-api-access-xnt8x\") pod \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " Sep 15 21:49:52.466736 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.466689 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls\") pod \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " Sep 15 21:49:52.466736 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.466725 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-metrics-certs\") pod \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\" (UID: \"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e\") " Sep 15 21:49:52.466906 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.466876 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" (UID: "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:49:52.468928 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.468895 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" (UID: "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:49:52.468928 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.468919 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" (UID: "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:49:52.469077 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.468952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-kube-api-access-xnt8x" (OuterVolumeSpecName: "kube-api-access-xnt8x") pod "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" (UID: "87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e"). InnerVolumeSpecName "kube-api-access-xnt8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:49:52.568444 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.568403 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.568444 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.568430 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.568444 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.568439 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnt8x\" (UniqueName: \"kubernetes.io/projected/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-kube-api-access-xnt8x\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.568444 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:52.568449 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:49:53.036241 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.036203 2575 generic.go:356] "Generic (PLEG): container finished" podID="87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" containerID="7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848" exitCode=0 Sep 15 21:49:53.036428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.036246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" event={"ID":"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e","Type":"ContainerDied","Data":"7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848"} Sep 15 21:49:53.036428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.036255 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" Sep 15 21:49:53.036428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.036289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp" event={"ID":"87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e","Type":"ContainerDied","Data":"e3b8bc5f1668e48d58d7ffc6a5a62482eed6ad32dd3dca0ee1dfe6a4e4f7343c"} Sep 15 21:49:53.036428 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.036309 2575 scope.go:117] "RemoveContainer" containerID="7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848" Sep 15 21:49:53.046052 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.046032 2575 scope.go:117] "RemoveContainer" containerID="7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848" Sep 15 21:49:53.046350 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:49:53.046328 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848\": container with ID starting with 7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848 not found: ID does not exist" containerID="7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848" Sep 15 21:49:53.046413 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.046361 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848"} err="failed to get container status \"7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848\": rpc error: code = NotFound desc = could not find container \"7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848\": container with ID starting with 7abcae090066efbda58d6599565df9dc56648005699f7ce2fbca067d68f2a848 not found: ID does not exist" Sep 15 21:49:53.057150 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.057128 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp"] Sep 15 21:49:53.060290 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.060272 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-459b305b-856bc55878-g28dp"] Sep 15 21:49:53.147616 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:49:53.147587 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" path="/var/lib/kubelet/pods/87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e/volumes" Sep 15 21:50:13.631565 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.631302 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q"] Sep 15 21:50:13.632045 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.631844 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" containerName="maas-api" Sep 15 21:50:13.632045 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.631863 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" containerName="maas-api" Sep 15 21:50:13.632045 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.631974 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fc2ada-77ce-4a3f-b13c-aa6b2e2b4d5e" containerName="maas-api" Sep 15 21:50:13.636959 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.636938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.639628 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.639608 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"openshift-service-ca.crt\"" Sep 15 21:50:13.639728 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.639610 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"kube-root-ca.crt\"" Sep 15 21:50:13.640418 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.640398 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"auth-test-model-ce3ab0-kserve-self-signed-certs\"" Sep 15 21:50:13.640531 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.640454 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"default-dockercfg-lxhpx\"" Sep 15 21:50:13.645540 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.645513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q"] Sep 15 21:50:13.767302 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.767272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-dshm\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.767487 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.767318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tmp-dir\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.767487 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.767409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tls-certs\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.767487 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.767474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-home\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.767643 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.767522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5snf\" (UniqueName: \"kubernetes.io/projected/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-kube-api-access-j5snf\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.767643 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.767575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-model-cache\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.868735 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.868702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tls-certs\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.868900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.868752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-home\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.868900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.868779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5snf\" (UniqueName: \"kubernetes.io/projected/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-kube-api-access-j5snf\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.868900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.868817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-model-cache\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.868900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.868854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-dshm\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.868900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.868896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tmp-dir\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.869205 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.869187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-home\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.869266 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.869241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-model-cache\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.869266 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.869260 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tmp-dir\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.871287 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.871263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-dshm\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.871525 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.871508 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tls-certs\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.876362 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.876337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5snf\" (UniqueName: \"kubernetes.io/projected/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-kube-api-access-j5snf\") pod \"auth-test-model-ce3ab0-kserve-64667fc55-rqg2q\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:13.948337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:13.948309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:14.219008 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:14.218984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q"] Sep 15 21:50:14.221126 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:50:14.221079 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12ef3ce1_21fa_410e_b96a_ac087bb6aac5.slice/crio-d4e5c4dc86c8559d686797b0c91d65557bd9101e9e1f97a64930dd2793706c89 WatchSource:0}: Error finding container d4e5c4dc86c8559d686797b0c91d65557bd9101e9e1f97a64930dd2793706c89: Status 404 returned error can't find the container with id d4e5c4dc86c8559d686797b0c91d65557bd9101e9e1f97a64930dd2793706c89 Sep 15 21:50:15.159247 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:15.159211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" event={"ID":"12ef3ce1-21fa-410e-b96a-ac087bb6aac5","Type":"ContainerStarted","Data":"9cc9052a6fc97dd781fbed73e06465cbf380dfcadb44c214e4e01766a9ab06e3"} Sep 15 21:50:15.159610 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:15.159255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" event={"ID":"12ef3ce1-21fa-410e-b96a-ac087bb6aac5","Type":"ContainerStarted","Data":"d4e5c4dc86c8559d686797b0c91d65557bd9101e9e1f97a64930dd2793706c89"} Sep 15 21:50:15.174450 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:15.174406 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" podStartSLOduration=2.17439196 podStartE2EDuration="2.17439196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:50:15.172796071 +0000 UTC m=+1464.512275521" watchObservedRunningTime="2026-09-15 21:50:15.17439196 +0000 UTC m=+1464.513871410" Sep 15 21:50:23.948832 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:23.948799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:23.949338 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:23.949068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:23.961486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:23.961457 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:50:24.224389 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:50:24.224313 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:51:59.836236 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.836198 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t"] Sep 15 21:51:59.840470 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.840443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:51:59.843128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.843084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-05557e-serving-cert\"" Sep 15 21:51:59.851477 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.851449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t"] Sep 15 21:51:59.908983 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.908943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85a0fe-10d7-441e-87de-dc46ba800b78-openshift-service-ca\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:51:59.909218 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.909007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-metrics-certs\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:51:59.909218 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.909165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssch\" (UniqueName: \"kubernetes.io/projected/7b85a0fe-10d7-441e-87de-dc46ba800b78-kube-api-access-fssch\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:51:59.909218 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:51:59.909205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-maas-api-tls\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.010539 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.010499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85a0fe-10d7-441e-87de-dc46ba800b78-openshift-service-ca\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.010718 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.010565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-metrics-certs\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.010718 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.010619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssch\" (UniqueName: \"kubernetes.io/projected/7b85a0fe-10d7-441e-87de-dc46ba800b78-kube-api-access-fssch\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.010718 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.010636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-maas-api-tls\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.011363 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.011255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85a0fe-10d7-441e-87de-dc46ba800b78-openshift-service-ca\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.013135 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.013113 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-maas-api-tls\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.013248 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.013171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-metrics-certs\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.018061 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.018035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssch\" (UniqueName: \"kubernetes.io/projected/7b85a0fe-10d7-441e-87de-dc46ba800b78-kube-api-access-fssch\") pod \"maas-api-e2e-api-a-05557e-757fcc85d6-59h9t\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.155172 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.155075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:00.434002 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.433972 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t"] Sep 15 21:52:00.435988 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:52:00.435955 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b85a0fe_10d7_441e_87de_dc46ba800b78.slice/crio-8b4c31ebd1f65c00c1eee7604169a1d8102381e8cd7c77d5201ec9a7c72ed4b2 WatchSource:0}: Error finding container 8b4c31ebd1f65c00c1eee7604169a1d8102381e8cd7c77d5201ec9a7c72ed4b2: Status 404 returned error can't find the container with id 8b4c31ebd1f65c00c1eee7604169a1d8102381e8cd7c77d5201ec9a7c72ed4b2 Sep 15 21:52:00.437512 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.437489 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:52:00.729648 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:00.729546 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" event={"ID":"7b85a0fe-10d7-441e-87de-dc46ba800b78","Type":"ContainerStarted","Data":"8b4c31ebd1f65c00c1eee7604169a1d8102381e8cd7c77d5201ec9a7c72ed4b2"} Sep 15 21:52:01.735996 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:01.735965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" event={"ID":"7b85a0fe-10d7-441e-87de-dc46ba800b78","Type":"ContainerStarted","Data":"e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808"} Sep 15 21:52:01.736500 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:01.736109 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:01.753971 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:01.753921 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" podStartSLOduration=2.187647615 podStartE2EDuration="2.753904996s" totalImagesPullingTime="566.257381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:01.752368621 +0000 UTC m=+1571.091848072" watchObservedRunningTime="2026-09-15 21:52:01.753904996 +0000 UTC m=+1571.093384447" Sep 15 21:52:07.747418 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:07.747389 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:52:34.313360 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.313328 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns"] Sep 15 21:52:34.317092 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.317072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.320966 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.320944 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-74da27-serving-cert\"" Sep 15 21:52:34.342733 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.342707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns"] Sep 15 21:52:34.460400 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.460363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70761b49-8e24-48fc-a5f7-06a46b7da7bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.460575 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.460412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.460575 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.460540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-metrics-certs\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.460659 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.460637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrqn\" (UniqueName: \"kubernetes.io/projected/70761b49-8e24-48fc-a5f7-06a46b7da7bc-kube-api-access-ktrqn\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.561946 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.561908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrqn\" (UniqueName: \"kubernetes.io/projected/70761b49-8e24-48fc-a5f7-06a46b7da7bc-kube-api-access-ktrqn\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.561946 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.561954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70761b49-8e24-48fc-a5f7-06a46b7da7bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.562252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.561992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.562252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.562093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-metrics-certs\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.562252 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:52:34.562232 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-serving-cert: secret "maas-api-e2e-api-b-74da27-serving-cert" not found Sep 15 21:52:34.562398 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:52:34.562330 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls podName:70761b49-8e24-48fc-a5f7-06a46b7da7bc nodeName:}" failed. No retries permitted until 2026-09-15 21:52:35.062290439 +0000 UTC m=+1604.401769885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls") pod "maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" (UID: "70761b49-8e24-48fc-a5f7-06a46b7da7bc") : secret "maas-api-e2e-api-b-74da27-serving-cert" not found Sep 15 21:52:34.562718 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.562696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70761b49-8e24-48fc-a5f7-06a46b7da7bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.564880 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.564829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-metrics-certs\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:34.572511 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:34.572487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrqn\" (UniqueName: \"kubernetes.io/projected/70761b49-8e24-48fc-a5f7-06a46b7da7bc-kube-api-access-ktrqn\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:35.067012 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:35.066954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:35.069808 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:35.069789 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-74da27-5c58d7d45-hrpns\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:35.228955 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:35.228920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:35.507492 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:35.507467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns"] Sep 15 21:52:35.509433 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:52:35.509406 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70761b49_8e24_48fc_a5f7_06a46b7da7bc.slice/crio-a420517161f9fd19129381c6dabef3fd97ee7b26b28a2bd2ec5381b985e6041b WatchSource:0}: Error finding container a420517161f9fd19129381c6dabef3fd97ee7b26b28a2bd2ec5381b985e6041b: Status 404 returned error can't find the container with id a420517161f9fd19129381c6dabef3fd97ee7b26b28a2bd2ec5381b985e6041b Sep 15 21:52:35.918834 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:35.918802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" event={"ID":"70761b49-8e24-48fc-a5f7-06a46b7da7bc","Type":"ContainerStarted","Data":"a420517161f9fd19129381c6dabef3fd97ee7b26b28a2bd2ec5381b985e6041b"} Sep 15 21:52:36.925651 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:36.925616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" event={"ID":"70761b49-8e24-48fc-a5f7-06a46b7da7bc","Type":"ContainerStarted","Data":"cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5"} Sep 15 21:52:36.926068 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:36.925761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:52:36.942351 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:36.942301 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" podStartSLOduration=2.45909524 podStartE2EDuration="2.942287179s" totalImagesPullingTime="483.191939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:36.940896374 +0000 UTC m=+1606.280375848" watchObservedRunningTime="2026-09-15 21:52:36.942287179 +0000 UTC m=+1606.281766631" Sep 15 21:52:42.936977 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:52:42.936891 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:53:18.342527 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.342463 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t"] Sep 15 21:53:18.343164 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.342744 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" podUID="7b85a0fe-10d7-441e-87de-dc46ba800b78" containerName="maas-api" containerID="cri-o://e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808" gracePeriod=30 Sep 15 21:53:18.611794 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.611768 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:53:18.695064 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.695029 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-metrics-certs\") pod \"7b85a0fe-10d7-441e-87de-dc46ba800b78\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " Sep 15 21:53:18.695278 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.695122 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fssch\" (UniqueName: \"kubernetes.io/projected/7b85a0fe-10d7-441e-87de-dc46ba800b78-kube-api-access-fssch\") pod \"7b85a0fe-10d7-441e-87de-dc46ba800b78\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " Sep 15 21:53:18.695278 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.695168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-maas-api-tls\") pod \"7b85a0fe-10d7-441e-87de-dc46ba800b78\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " Sep 15 21:53:18.695278 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.695209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85a0fe-10d7-441e-87de-dc46ba800b78-openshift-service-ca\") pod \"7b85a0fe-10d7-441e-87de-dc46ba800b78\" (UID: \"7b85a0fe-10d7-441e-87de-dc46ba800b78\") " Sep 15 21:53:18.695657 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.695625 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b85a0fe-10d7-441e-87de-dc46ba800b78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b85a0fe-10d7-441e-87de-dc46ba800b78" (UID: "7b85a0fe-10d7-441e-87de-dc46ba800b78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:53:18.697886 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.697857 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b85a0fe-10d7-441e-87de-dc46ba800b78-kube-api-access-fssch" (OuterVolumeSpecName: "kube-api-access-fssch") pod "7b85a0fe-10d7-441e-87de-dc46ba800b78" (UID: "7b85a0fe-10d7-441e-87de-dc46ba800b78"). InnerVolumeSpecName "kube-api-access-fssch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:53:18.698140 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.698114 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b85a0fe-10d7-441e-87de-dc46ba800b78" (UID: "7b85a0fe-10d7-441e-87de-dc46ba800b78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:53:18.698250 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.698135 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b85a0fe-10d7-441e-87de-dc46ba800b78" (UID: "7b85a0fe-10d7-441e-87de-dc46ba800b78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:53:18.796167 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.796137 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:18.796167 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.796164 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssch\" (UniqueName: \"kubernetes.io/projected/7b85a0fe-10d7-441e-87de-dc46ba800b78-kube-api-access-fssch\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:18.796167 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.796172 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85a0fe-10d7-441e-87de-dc46ba800b78-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:18.796394 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:18.796181 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85a0fe-10d7-441e-87de-dc46ba800b78-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:19.155542 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.155515 2575 generic.go:356] "Generic (PLEG): container finished" podID="7b85a0fe-10d7-441e-87de-dc46ba800b78" containerID="e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808" exitCode=0 Sep 15 21:53:19.155686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.155575 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" Sep 15 21:53:19.155686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.155599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" event={"ID":"7b85a0fe-10d7-441e-87de-dc46ba800b78","Type":"ContainerDied","Data":"e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808"} Sep 15 21:53:19.155686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.155662 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t" event={"ID":"7b85a0fe-10d7-441e-87de-dc46ba800b78","Type":"ContainerDied","Data":"8b4c31ebd1f65c00c1eee7604169a1d8102381e8cd7c77d5201ec9a7c72ed4b2"} Sep 15 21:53:19.155874 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.155692 2575 scope.go:117] "RemoveContainer" containerID="e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808" Sep 15 21:53:19.165236 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.165219 2575 scope.go:117] "RemoveContainer" containerID="e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808" Sep 15 21:53:19.165515 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:53:19.165497 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808\": container with ID starting with e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808 not found: ID does not exist" containerID="e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808" Sep 15 21:53:19.165584 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.165522 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808"} err="failed to get container status \"e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808\": rpc error: code = NotFound desc = could not find container \"e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808\": container with ID starting with e8b04e406880c324f1edc2c8eda8eae38f2b7a5dcaead5abaa2b7179e80b9808 not found: ID does not exist" Sep 15 21:53:19.173713 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.173691 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t"] Sep 15 21:53:19.180610 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:19.180579 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05557e-757fcc85d6-59h9t"] Sep 15 21:53:21.147404 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:21.147370 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b85a0fe-10d7-441e-87de-dc46ba800b78" path="/var/lib/kubelet/pods/7b85a0fe-10d7-441e-87de-dc46ba800b78/volumes" Sep 15 21:53:26.195250 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:26.195216 2575 generic.go:356] "Generic (PLEG): container finished" podID="2020d892-df37-466b-8134-3093e91d366b" containerID="ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4" exitCode=137 Sep 15 21:53:26.195673 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:26.195265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" event={"ID":"2020d892-df37-466b-8134-3093e91d366b","Type":"ContainerDied","Data":"ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4"} Sep 15 21:53:26.195673 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:26.195574 2575 scope.go:117] "RemoveContainer" containerID="ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4" Sep 15 21:53:26.203748 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:26.203728 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:53:27.202893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:27.202859 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" event={"ID":"2020d892-df37-466b-8134-3093e91d366b","Type":"ContainerStarted","Data":"842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8"} Sep 15 21:53:27.203283 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:27.203020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:53:38.210494 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:38.210464 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 21:53:40.976611 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:40.976570 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns"] Sep 15 21:53:40.977646 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:40.977297 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" podUID="70761b49-8e24-48fc-a5f7-06a46b7da7bc" containerName="maas-api" containerID="cri-o://cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5" gracePeriod=30 Sep 15 21:53:41.236002 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.235949 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:53:41.281926 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.281892 2575 generic.go:356] "Generic (PLEG): container finished" podID="70761b49-8e24-48fc-a5f7-06a46b7da7bc" containerID="cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5" exitCode=0 Sep 15 21:53:41.282121 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.281944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" event={"ID":"70761b49-8e24-48fc-a5f7-06a46b7da7bc","Type":"ContainerDied","Data":"cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5"} Sep 15 21:53:41.282121 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.281954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" Sep 15 21:53:41.282121 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.281998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns" event={"ID":"70761b49-8e24-48fc-a5f7-06a46b7da7bc","Type":"ContainerDied","Data":"a420517161f9fd19129381c6dabef3fd97ee7b26b28a2bd2ec5381b985e6041b"} Sep 15 21:53:41.282121 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.282028 2575 scope.go:117] "RemoveContainer" containerID="cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5" Sep 15 21:53:41.293315 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.293295 2575 scope.go:117] "RemoveContainer" containerID="cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5" Sep 15 21:53:41.293602 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:53:41.293578 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5\": container with ID starting with cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5 not found: ID does not exist" containerID="cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5" Sep 15 21:53:41.293657 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.293615 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5"} err="failed to get container status \"cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5\": rpc error: code = NotFound desc = could not find container \"cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5\": container with ID starting with cbb4e15830cf42d556761e73b13200ca2b873904c89cf279df862520c17583b5 not found: ID does not exist" Sep 15 21:53:41.315986 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.315959 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls\") pod \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " Sep 15 21:53:41.316157 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.315998 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70761b49-8e24-48fc-a5f7-06a46b7da7bc-openshift-service-ca\") pod \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " Sep 15 21:53:41.316157 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.316019 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktrqn\" (UniqueName: \"kubernetes.io/projected/70761b49-8e24-48fc-a5f7-06a46b7da7bc-kube-api-access-ktrqn\") pod \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " Sep 15 21:53:41.316300 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.316164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-metrics-certs\") pod \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\" (UID: \"70761b49-8e24-48fc-a5f7-06a46b7da7bc\") " Sep 15 21:53:41.316399 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.316378 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70761b49-8e24-48fc-a5f7-06a46b7da7bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70761b49-8e24-48fc-a5f7-06a46b7da7bc" (UID: "70761b49-8e24-48fc-a5f7-06a46b7da7bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:53:41.316603 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.316578 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70761b49-8e24-48fc-a5f7-06a46b7da7bc-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:41.318357 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.318327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70761b49-8e24-48fc-a5f7-06a46b7da7bc" (UID: "70761b49-8e24-48fc-a5f7-06a46b7da7bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:53:41.318457 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.318393 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70761b49-8e24-48fc-a5f7-06a46b7da7bc-kube-api-access-ktrqn" (OuterVolumeSpecName: "kube-api-access-ktrqn") pod "70761b49-8e24-48fc-a5f7-06a46b7da7bc" (UID: "70761b49-8e24-48fc-a5f7-06a46b7da7bc"). InnerVolumeSpecName "kube-api-access-ktrqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:53:41.318935 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.318909 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70761b49-8e24-48fc-a5f7-06a46b7da7bc" (UID: "70761b49-8e24-48fc-a5f7-06a46b7da7bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:53:41.417623 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.417585 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:41.417773 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.417626 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktrqn\" (UniqueName: \"kubernetes.io/projected/70761b49-8e24-48fc-a5f7-06a46b7da7bc-kube-api-access-ktrqn\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:41.417773 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.417643 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70761b49-8e24-48fc-a5f7-06a46b7da7bc-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:41.605552 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.605521 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns"] Sep 15 21:53:41.608465 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:41.608442 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74da27-5c58d7d45-hrpns"] Sep 15 21:53:43.147515 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:43.147470 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70761b49-8e24-48fc-a5f7-06a46b7da7bc" path="/var/lib/kubelet/pods/70761b49-8e24-48fc-a5f7-06a46b7da7bc/volumes" Sep 15 21:53:58.534973 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.534930 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59"] Sep 15 21:53:58.535560 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.535246 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" podUID="26c09de5-9e48-4b1d-8810-de528c541431" containerName="maas-api" containerID="cri-o://4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c" gracePeriod=30 Sep 15 21:53:58.803153 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.803067 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:53:58.895837 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.895800 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") pod \"26c09de5-9e48-4b1d-8810-de528c541431\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " Sep 15 21:53:58.896033 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.895867 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c09de5-9e48-4b1d-8810-de528c541431-openshift-service-ca\") pod \"26c09de5-9e48-4b1d-8810-de528c541431\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " Sep 15 21:53:58.896033 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.896004 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7smsr\" (UniqueName: \"kubernetes.io/projected/26c09de5-9e48-4b1d-8810-de528c541431-kube-api-access-7smsr\") pod \"26c09de5-9e48-4b1d-8810-de528c541431\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " Sep 15 21:53:58.896202 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.896166 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-metrics-certs\") pod \"26c09de5-9e48-4b1d-8810-de528c541431\" (UID: \"26c09de5-9e48-4b1d-8810-de528c541431\") " Sep 15 21:53:58.896264 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.896242 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c09de5-9e48-4b1d-8810-de528c541431-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26c09de5-9e48-4b1d-8810-de528c541431" (UID: "26c09de5-9e48-4b1d-8810-de528c541431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:53:58.896543 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.896519 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c09de5-9e48-4b1d-8810-de528c541431-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:58.898197 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.898167 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26c09de5-9e48-4b1d-8810-de528c541431" (UID: "26c09de5-9e48-4b1d-8810-de528c541431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:53:58.898702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.898682 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26c09de5-9e48-4b1d-8810-de528c541431" (UID: "26c09de5-9e48-4b1d-8810-de528c541431"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:53:58.898778 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.898756 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c09de5-9e48-4b1d-8810-de528c541431-kube-api-access-7smsr" (OuterVolumeSpecName: "kube-api-access-7smsr") pod "26c09de5-9e48-4b1d-8810-de528c541431" (UID: "26c09de5-9e48-4b1d-8810-de528c541431"). InnerVolumeSpecName "kube-api-access-7smsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:53:58.998272 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.998224 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:58.998272 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.998265 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7smsr\" (UniqueName: \"kubernetes.io/projected/26c09de5-9e48-4b1d-8810-de528c541431-kube-api-access-7smsr\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:58.998453 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:58.998284 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c09de5-9e48-4b1d-8810-de528c541431-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:53:59.380884 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.380846 2575 generic.go:356] "Generic (PLEG): container finished" podID="26c09de5-9e48-4b1d-8810-de528c541431" containerID="4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c" exitCode=0 Sep 15 21:53:59.381066 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.380898 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" Sep 15 21:53:59.381066 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.380911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" event={"ID":"26c09de5-9e48-4b1d-8810-de528c541431","Type":"ContainerDied","Data":"4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c"} Sep 15 21:53:59.381066 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.380965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59" event={"ID":"26c09de5-9e48-4b1d-8810-de528c541431","Type":"ContainerDied","Data":"fb2caa2930f18c10d01f16ca81dc776a03124503f0f026351ae5f2fb95c30c9e"} Sep 15 21:53:59.381066 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.380995 2575 scope.go:117] "RemoveContainer" containerID="4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c" Sep 15 21:53:59.390789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.390769 2575 scope.go:117] "RemoveContainer" containerID="4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c" Sep 15 21:53:59.391114 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:53:59.391073 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c\": container with ID starting with 4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c not found: ID does not exist" containerID="4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c" Sep 15 21:53:59.391176 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.391119 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c"} err="failed to get container status \"4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c\": rpc error: code = NotFound desc = could not find container \"4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c\": container with ID starting with 4626745ccd1115ff4cba0f942ecd28a7baceb9ed20a2ec89825f74e02fef5e2c not found: ID does not exist" Sep 15 21:53:59.397636 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.397612 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59"] Sep 15 21:53:59.399742 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:53:59.399723 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-850f22-5fbcc5f75-7kx59"] Sep 15 21:54:01.148034 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:01.148004 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c09de5-9e48-4b1d-8810-de528c541431" path="/var/lib/kubelet/pods/26c09de5-9e48-4b1d-8810-de528c541431/volumes" Sep 15 21:54:16.133802 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.133715 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh"] Sep 15 21:54:16.134236 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.134020 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" podUID="daa2c63c-388e-4045-bca3-e904dc2b90ff" containerName="maas-api" containerID="cri-o://377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c" gracePeriod=30 Sep 15 21:54:16.394540 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.394480 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:54:16.461244 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.461215 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbzjw\" (UniqueName: \"kubernetes.io/projected/daa2c63c-388e-4045-bca3-e904dc2b90ff-kube-api-access-tbzjw\") pod \"daa2c63c-388e-4045-bca3-e904dc2b90ff\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " Sep 15 21:54:16.461420 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.461322 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa2c63c-388e-4045-bca3-e904dc2b90ff-openshift-service-ca\") pod \"daa2c63c-388e-4045-bca3-e904dc2b90ff\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " Sep 15 21:54:16.461420 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.461358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-metrics-certs\") pod \"daa2c63c-388e-4045-bca3-e904dc2b90ff\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " Sep 15 21:54:16.461420 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.461406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-maas-api-tls\") pod \"daa2c63c-388e-4045-bca3-e904dc2b90ff\" (UID: \"daa2c63c-388e-4045-bca3-e904dc2b90ff\") " Sep 15 21:54:16.461675 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.461647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa2c63c-388e-4045-bca3-e904dc2b90ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "daa2c63c-388e-4045-bca3-e904dc2b90ff" (UID: "daa2c63c-388e-4045-bca3-e904dc2b90ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:54:16.463612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.463584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa2c63c-388e-4045-bca3-e904dc2b90ff-kube-api-access-tbzjw" (OuterVolumeSpecName: "kube-api-access-tbzjw") pod "daa2c63c-388e-4045-bca3-e904dc2b90ff" (UID: "daa2c63c-388e-4045-bca3-e904dc2b90ff"). InnerVolumeSpecName "kube-api-access-tbzjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:16.463724 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.463610 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "daa2c63c-388e-4045-bca3-e904dc2b90ff" (UID: "daa2c63c-388e-4045-bca3-e904dc2b90ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:16.463724 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.463694 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "daa2c63c-388e-4045-bca3-e904dc2b90ff" (UID: "daa2c63c-388e-4045-bca3-e904dc2b90ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:16.471739 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.471713 2575 generic.go:356] "Generic (PLEG): container finished" podID="daa2c63c-388e-4045-bca3-e904dc2b90ff" containerID="377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c" exitCode=0 Sep 15 21:54:16.471855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.471774 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" Sep 15 21:54:16.471855 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.471802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" event={"ID":"daa2c63c-388e-4045-bca3-e904dc2b90ff","Type":"ContainerDied","Data":"377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c"} Sep 15 21:54:16.471978 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.471865 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh" event={"ID":"daa2c63c-388e-4045-bca3-e904dc2b90ff","Type":"ContainerDied","Data":"bbbf8f24628893e2c90f62eb51a84653cdad232fa3700c31245fef76dbf30b3b"} Sep 15 21:54:16.471978 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.471897 2575 scope.go:117] "RemoveContainer" containerID="377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c" Sep 15 21:54:16.488658 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.488635 2575 scope.go:117] "RemoveContainer" containerID="377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c" Sep 15 21:54:16.489004 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:54:16.488983 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c\": container with ID starting with 377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c not found: ID does not exist" containerID="377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c" Sep 15 21:54:16.489074 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.489015 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c"} err="failed to get container status \"377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c\": rpc error: code = NotFound desc = could not find container \"377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c\": container with ID starting with 377c3aa7dbe26cba3547b66e18adddf2866b3af9b2fb10bcd70b198d06f8668c not found: ID does not exist" Sep 15 21:54:16.498285 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.498253 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh"] Sep 15 21:54:16.505075 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.505049 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8c1d2a-55c4597cc6-vsgfh"] Sep 15 21:54:16.563036 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.563009 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa2c63c-388e-4045-bca3-e904dc2b90ff-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:16.563036 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.563034 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:16.563036 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.563044 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa2c63c-388e-4045-bca3-e904dc2b90ff-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:16.563288 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:16.563053 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbzjw\" (UniqueName: \"kubernetes.io/projected/daa2c63c-388e-4045-bca3-e904dc2b90ff-kube-api-access-tbzjw\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:17.148111 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:17.148067 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa2c63c-388e-4045-bca3-e904dc2b90ff" path="/var/lib/kubelet/pods/daa2c63c-388e-4045-bca3-e904dc2b90ff/volumes" Sep 15 21:54:32.512643 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:32.512607 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2"] Sep 15 21:54:32.517856 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:32.517832 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp"] Sep 15 21:54:32.538686 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:32.538658 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9"] Sep 15 21:54:32.546357 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:32.546062 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq"] Sep 15 21:54:43.227046 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.227011 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv"] Sep 15 21:54:43.227556 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.227295 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" podUID="8b991f35-71f9-4422-9896-068e62662be0" containerName="maas-api" containerID="cri-o://71d1b61cb5167dda69c040d975d0df05b6ddd5d0bd468798d0b7927627ba69a5" gracePeriod=30 Sep 15 21:54:43.618467 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.618433 2575 generic.go:356] "Generic (PLEG): container finished" podID="8b991f35-71f9-4422-9896-068e62662be0" containerID="71d1b61cb5167dda69c040d975d0df05b6ddd5d0bd468798d0b7927627ba69a5" exitCode=0 Sep 15 21:54:43.618610 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.618509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" event={"ID":"8b991f35-71f9-4422-9896-068e62662be0","Type":"ContainerDied","Data":"71d1b61cb5167dda69c040d975d0df05b6ddd5d0bd468798d0b7927627ba69a5"} Sep 15 21:54:43.694415 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.694388 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:54:43.838593 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.838518 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls\") pod \"8b991f35-71f9-4422-9896-068e62662be0\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " Sep 15 21:54:43.838593 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.838560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b991f35-71f9-4422-9896-068e62662be0-openshift-service-ca\") pod \"8b991f35-71f9-4422-9896-068e62662be0\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " Sep 15 21:54:43.838781 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.838609 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-metrics-certs\") pod \"8b991f35-71f9-4422-9896-068e62662be0\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " Sep 15 21:54:43.838781 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.838641 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwr9v\" (UniqueName: \"kubernetes.io/projected/8b991f35-71f9-4422-9896-068e62662be0-kube-api-access-cwr9v\") pod \"8b991f35-71f9-4422-9896-068e62662be0\" (UID: \"8b991f35-71f9-4422-9896-068e62662be0\") " Sep 15 21:54:43.838991 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.838957 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b991f35-71f9-4422-9896-068e62662be0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b991f35-71f9-4422-9896-068e62662be0" (UID: "8b991f35-71f9-4422-9896-068e62662be0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:54:43.841036 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.841002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b991f35-71f9-4422-9896-068e62662be0" (UID: "8b991f35-71f9-4422-9896-068e62662be0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:43.841181 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.841091 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b991f35-71f9-4422-9896-068e62662be0" (UID: "8b991f35-71f9-4422-9896-068e62662be0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:43.841181 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.841130 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b991f35-71f9-4422-9896-068e62662be0-kube-api-access-cwr9v" (OuterVolumeSpecName: "kube-api-access-cwr9v") pod "8b991f35-71f9-4422-9896-068e62662be0" (UID: "8b991f35-71f9-4422-9896-068e62662be0"). InnerVolumeSpecName "kube-api-access-cwr9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:43.940506 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.940476 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:43.940506 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.940503 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b991f35-71f9-4422-9896-068e62662be0-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:43.940678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.940514 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b991f35-71f9-4422-9896-068e62662be0-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:43.940678 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:43.940523 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwr9v\" (UniqueName: \"kubernetes.io/projected/8b991f35-71f9-4422-9896-068e62662be0-kube-api-access-cwr9v\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:44.625139 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:44.625093 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" Sep 15 21:54:44.625139 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:44.625121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv" event={"ID":"8b991f35-71f9-4422-9896-068e62662be0","Type":"ContainerDied","Data":"25e5753ce37d83bae45160ea8a28dc6ed23637737d961c1dce05ede8b7481c19"} Sep 15 21:54:44.625662 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:44.625204 2575 scope.go:117] "RemoveContainer" containerID="71d1b61cb5167dda69c040d975d0df05b6ddd5d0bd468798d0b7927627ba69a5" Sep 15 21:54:44.650504 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:44.650482 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv"] Sep 15 21:54:44.656397 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:44.656373 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-360e43-58bcd96fcb-5gklv"] Sep 15 21:54:45.147888 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:45.147845 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b991f35-71f9-4422-9896-068e62662be0" path="/var/lib/kubelet/pods/8b991f35-71f9-4422-9896-068e62662be0/volumes" Sep 15 21:54:47.533349 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.533302 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" podUID="3c72862e-2fa9-463b-9d1b-c694e803c03e" containerName="main" containerID="cri-o://44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c" gracePeriod=45 Sep 15 21:54:47.537450 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.537410 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" podUID="d7eede26-79bf-432b-b2fb-57973c63749f" containerName="main" containerID="cri-o://9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab" gracePeriod=45 Sep 15 21:54:47.565476 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.565444 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" podUID="3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" containerName="main" containerID="cri-o://166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8" gracePeriod=45 Sep 15 21:54:47.566053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.566024 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" podUID="39bad9c6-6934-4522-b065-473d9891f551" containerName="main" containerID="cri-o://33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b" gracePeriod=45 Sep 15 21:54:47.859583 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.859555 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:54:47.881799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.881719 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:54:47.891012 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.890976 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:54:47.894531 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.894506 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:54:47.978032 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978005 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-model-cache\") pod \"3c72862e-2fa9-463b-9d1b-c694e803c03e\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978057 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-dshm\") pod \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978078 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-dshm\") pod \"3c72862e-2fa9-463b-9d1b-c694e803c03e\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978128 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72862e-2fa9-463b-9d1b-c694e803c03e-tls-certs\") pod \"3c72862e-2fa9-463b-9d1b-c694e803c03e\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978146 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-tmp-dir\") pod \"3c72862e-2fa9-463b-9d1b-c694e803c03e\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978196 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-home\") pod \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978225 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tls-certs\") pod \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " Sep 15 21:54:47.978252 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dm7v\" (UniqueName: \"kubernetes.io/projected/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-kube-api-access-4dm7v\") pod \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " Sep 15 21:54:47.978640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978292 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-home\") pod \"3c72862e-2fa9-463b-9d1b-c694e803c03e\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " Sep 15 21:54:47.978640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978320 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-model-cache\") pod \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " Sep 15 21:54:47.978640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tmp-dir\") pod \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\" (UID: \"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8\") " Sep 15 21:54:47.978640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978366 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c72862e-2fa9-463b-9d1b-c694e803c03e" (UID: "3c72862e-2fa9-463b-9d1b-c694e803c03e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.978640 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978380 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzlvl\" (UniqueName: \"kubernetes.io/projected/3c72862e-2fa9-463b-9d1b-c694e803c03e-kube-api-access-pzlvl\") pod \"3c72862e-2fa9-463b-9d1b-c694e803c03e\" (UID: \"3c72862e-2fa9-463b-9d1b-c694e803c03e\") " Sep 15 21:54:47.979006 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-home" (OuterVolumeSpecName: "home") pod "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" (UID: "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.979184 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.979158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c72862e-2fa9-463b-9d1b-c694e803c03e" (UID: "3c72862e-2fa9-463b-9d1b-c694e803c03e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.979253 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.979168 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" (UID: "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.979369 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.978968 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:47.979435 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.979380 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-home" (OuterVolumeSpecName: "home") pod "3c72862e-2fa9-463b-9d1b-c694e803c03e" (UID: "3c72862e-2fa9-463b-9d1b-c694e803c03e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.979682 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.979646 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" (UID: "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.980596 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.980574 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-dshm" (OuterVolumeSpecName: "dshm") pod "3c72862e-2fa9-463b-9d1b-c694e803c03e" (UID: "3c72862e-2fa9-463b-9d1b-c694e803c03e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.980932 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.980904 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c72862e-2fa9-463b-9d1b-c694e803c03e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c72862e-2fa9-463b-9d1b-c694e803c03e" (UID: "3c72862e-2fa9-463b-9d1b-c694e803c03e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:47.981038 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.981022 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c72862e-2fa9-463b-9d1b-c694e803c03e-kube-api-access-pzlvl" (OuterVolumeSpecName: "kube-api-access-pzlvl") pod "3c72862e-2fa9-463b-9d1b-c694e803c03e" (UID: "3c72862e-2fa9-463b-9d1b-c694e803c03e"). InnerVolumeSpecName "kube-api-access-pzlvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:47.981210 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.981191 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-dshm" (OuterVolumeSpecName: "dshm") pod "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" (UID: "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:47.981664 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.981647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-kube-api-access-4dm7v" (OuterVolumeSpecName: "kube-api-access-4dm7v") pod "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" (UID: "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8"). InnerVolumeSpecName "kube-api-access-4dm7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:47.981664 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:47.981651 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" (UID: "3fec278a-d859-42d9-bdf9-d9fdcb5e82d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:48.080489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080408 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-dshm\") pod \"d7eede26-79bf-432b-b2fb-57973c63749f\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " Sep 15 21:54:48.080489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080451 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-tmp-dir\") pod \"39bad9c6-6934-4522-b065-473d9891f551\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " Sep 15 21:54:48.080489 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080477 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-tmp-dir\") pod \"d7eede26-79bf-432b-b2fb-57973c63749f\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-home\") pod \"d7eede26-79bf-432b-b2fb-57973c63749f\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080542 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djfrh\" (UniqueName: \"kubernetes.io/projected/39bad9c6-6934-4522-b065-473d9891f551-kube-api-access-djfrh\") pod \"39bad9c6-6934-4522-b065-473d9891f551\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080615 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-model-cache\") pod \"39bad9c6-6934-4522-b065-473d9891f551\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080654 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bad9c6-6934-4522-b065-473d9891f551-tls-certs\") pod \"39bad9c6-6934-4522-b065-473d9891f551\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080683 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7eede26-79bf-432b-b2fb-57973c63749f-tls-certs\") pod \"d7eede26-79bf-432b-b2fb-57973c63749f\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080736 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-model-cache\") pod \"d7eede26-79bf-432b-b2fb-57973c63749f\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080756 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7eede26-79bf-432b-b2fb-57973c63749f" (UID: "d7eede26-79bf-432b-b2fb-57973c63749f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.080782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-home\") pod \"39bad9c6-6934-4522-b065-473d9891f551\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " Sep 15 21:54:48.081257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080737 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39bad9c6-6934-4522-b065-473d9891f551" (UID: "39bad9c6-6934-4522-b065-473d9891f551"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.081257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-dshm\") pod \"39bad9c6-6934-4522-b065-473d9891f551\" (UID: \"39bad9c6-6934-4522-b065-473d9891f551\") " Sep 15 21:54:48.081257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080802 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-home" (OuterVolumeSpecName: "home") pod "d7eede26-79bf-432b-b2fb-57973c63749f" (UID: "d7eede26-79bf-432b-b2fb-57973c63749f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.081257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080828 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsgwq\" (UniqueName: \"kubernetes.io/projected/d7eede26-79bf-432b-b2fb-57973c63749f-kube-api-access-tsgwq\") pod \"d7eede26-79bf-432b-b2fb-57973c63749f\" (UID: \"d7eede26-79bf-432b-b2fb-57973c63749f\") " Sep 15 21:54:48.081257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.080991 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7eede26-79bf-432b-b2fb-57973c63749f" (UID: "d7eede26-79bf-432b-b2fb-57973c63749f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.081257 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081156 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-home" (OuterVolumeSpecName: "home") pod "39bad9c6-6934-4522-b065-473d9891f551" (UID: "39bad9c6-6934-4522-b065-473d9891f551"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081291 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-model-cache" (OuterVolumeSpecName: "model-cache") pod "39bad9c6-6934-4522-b065-473d9891f551" (UID: "39bad9c6-6934-4522-b065-473d9891f551"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081409 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081459 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081475 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081491 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dm7v\" (UniqueName: \"kubernetes.io/projected/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-kube-api-access-4dm7v\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081528 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081542 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081554 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081566 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081600 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzlvl\" (UniqueName: \"kubernetes.io/projected/3c72862e-2fa9-463b-9d1b-c694e803c03e-kube-api-access-pzlvl\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081614 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081626 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081638 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081650 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081681 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081696 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72862e-2fa9-463b-9d1b-c694e803c03e-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.081799 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.081708 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72862e-2fa9-463b-9d1b-c694e803c03e-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.083143 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.083114 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-dshm" (OuterVolumeSpecName: "dshm") pod "d7eede26-79bf-432b-b2fb-57973c63749f" (UID: "d7eede26-79bf-432b-b2fb-57973c63749f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.083502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.083479 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-dshm" (OuterVolumeSpecName: "dshm") pod "39bad9c6-6934-4522-b065-473d9891f551" (UID: "39bad9c6-6934-4522-b065-473d9891f551"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:48.083604 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.083581 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bad9c6-6934-4522-b065-473d9891f551-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39bad9c6-6934-4522-b065-473d9891f551" (UID: "39bad9c6-6934-4522-b065-473d9891f551"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:48.083777 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.083759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7eede26-79bf-432b-b2fb-57973c63749f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7eede26-79bf-432b-b2fb-57973c63749f" (UID: "d7eede26-79bf-432b-b2fb-57973c63749f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:54:48.083905 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.083881 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bad9c6-6934-4522-b065-473d9891f551-kube-api-access-djfrh" (OuterVolumeSpecName: "kube-api-access-djfrh") pod "39bad9c6-6934-4522-b065-473d9891f551" (UID: "39bad9c6-6934-4522-b065-473d9891f551"). InnerVolumeSpecName "kube-api-access-djfrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:48.084363 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.084346 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7eede26-79bf-432b-b2fb-57973c63749f-kube-api-access-tsgwq" (OuterVolumeSpecName: "kube-api-access-tsgwq") pod "d7eede26-79bf-432b-b2fb-57973c63749f" (UID: "d7eede26-79bf-432b-b2fb-57973c63749f"). InnerVolumeSpecName "kube-api-access-tsgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:48.182830 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182800 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bad9c6-6934-4522-b065-473d9891f551-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.182830 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182825 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7eede26-79bf-432b-b2fb-57973c63749f-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.182830 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182833 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.183034 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182842 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsgwq\" (UniqueName: \"kubernetes.io/projected/d7eede26-79bf-432b-b2fb-57973c63749f-kube-api-access-tsgwq\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.183034 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182852 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7eede26-79bf-432b-b2fb-57973c63749f-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.183034 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182860 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djfrh\" (UniqueName: \"kubernetes.io/projected/39bad9c6-6934-4522-b065-473d9891f551-kube-api-access-djfrh\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.183034 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.182868 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bad9c6-6934-4522-b065-473d9891f551-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:54:48.648067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.648030 2575 generic.go:356] "Generic (PLEG): container finished" podID="39bad9c6-6934-4522-b065-473d9891f551" containerID="33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b" exitCode=0 Sep 15 21:54:48.648513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.648082 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" Sep 15 21:54:48.648513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.648130 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" event={"ID":"39bad9c6-6934-4522-b065-473d9891f551","Type":"ContainerDied","Data":"33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b"} Sep 15 21:54:48.648513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.648175 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq" event={"ID":"39bad9c6-6934-4522-b065-473d9891f551","Type":"ContainerDied","Data":"223a426ac6c2d28a4b6236bc53c7639c20362e393e7494042ba72c59a9deb1a4"} Sep 15 21:54:48.648513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.648198 2575 scope.go:117] "RemoveContainer" containerID="33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b" Sep 15 21:54:48.649656 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.649465 2575 generic.go:356] "Generic (PLEG): container finished" podID="3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" containerID="166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8" exitCode=0 Sep 15 21:54:48.649656 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.649515 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" Sep 15 21:54:48.649656 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.649549 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" event={"ID":"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8","Type":"ContainerDied","Data":"166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8"} Sep 15 21:54:48.649656 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.649588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9" event={"ID":"3fec278a-d859-42d9-bdf9-d9fdcb5e82d8","Type":"ContainerDied","Data":"691da5711bec715d40ce9be2a9fc9125a8659dd9a53d79c7e8c454b15393e7f7"} Sep 15 21:54:48.651202 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.651175 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7eede26-79bf-432b-b2fb-57973c63749f" containerID="9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab" exitCode=0 Sep 15 21:54:48.651336 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.651200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" event={"ID":"d7eede26-79bf-432b-b2fb-57973c63749f","Type":"ContainerDied","Data":"9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab"} Sep 15 21:54:48.651336 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.651221 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" Sep 15 21:54:48.651336 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.651236 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp" event={"ID":"d7eede26-79bf-432b-b2fb-57973c63749f","Type":"ContainerDied","Data":"c4fb65ba932549b3fcb091c3c34a6c0e72818b99832c5e26d36059b40f075a38"} Sep 15 21:54:48.653156 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.653131 2575 generic.go:356] "Generic (PLEG): container finished" podID="3c72862e-2fa9-463b-9d1b-c694e803c03e" containerID="44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c" exitCode=0 Sep 15 21:54:48.653279 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.653164 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" event={"ID":"3c72862e-2fa9-463b-9d1b-c694e803c03e","Type":"ContainerDied","Data":"44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c"} Sep 15 21:54:48.653279 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.653171 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" Sep 15 21:54:48.653279 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.653188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2" event={"ID":"3c72862e-2fa9-463b-9d1b-c694e803c03e","Type":"ContainerDied","Data":"395090591b87395df82854d32e2838330948be41eddc7a9a5ed717c7bd9e31b7"} Sep 15 21:54:48.660187 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.660171 2575 scope.go:117] "RemoveContainer" containerID="33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b" Sep 15 21:54:48.660467 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:54:48.660448 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b\": container with ID starting with 33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b not found: ID does not exist" containerID="33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b" Sep 15 21:54:48.660590 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.660474 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b"} err="failed to get container status \"33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b\": rpc error: code = NotFound desc = could not find container \"33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b\": container with ID starting with 33cfe3be83654ec22669eb0720f4364d10bb3d1ef443ad77f57cce940bcc9b5b not found: ID does not exist" Sep 15 21:54:48.660590 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.660489 2575 scope.go:117] "RemoveContainer" containerID="166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8" Sep 15 21:54:48.669613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.669595 2575 scope.go:117] "RemoveContainer" containerID="166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8" Sep 15 21:54:48.669868 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:54:48.669852 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8\": container with ID starting with 166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8 not found: ID does not exist" containerID="166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8" Sep 15 21:54:48.669958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.669874 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8"} err="failed to get container status \"166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8\": rpc error: code = NotFound desc = could not find container \"166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8\": container with ID starting with 166e5ee1759527932c903da875b1ccfa575c9f5349d216480993164eb5e115c8 not found: ID does not exist" Sep 15 21:54:48.669958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.669890 2575 scope.go:117] "RemoveContainer" containerID="9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab" Sep 15 21:54:48.674642 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.674614 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq"] Sep 15 21:54:48.678980 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.678951 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/simulator-w0-360e43-kserve-6649cc64cb-q5rkq"] Sep 15 21:54:48.679671 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.679657 2575 scope.go:117] "RemoveContainer" containerID="9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab" Sep 15 21:54:48.679957 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:54:48.679940 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab\": container with ID starting with 9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab not found: ID does not exist" containerID="9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab" Sep 15 21:54:48.680014 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.679963 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab"} err="failed to get container status \"9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab\": rpc error: code = NotFound desc = could not find container \"9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab\": container with ID starting with 9c1cfaad8db10dfccd8a4b57a58a2bb267c09cfb757aefae7f05d6f12054b6ab not found: ID does not exist" Sep 15 21:54:48.680014 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.679982 2575 scope.go:117] "RemoveContainer" containerID="44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c" Sep 15 21:54:48.688164 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.688144 2575 scope.go:117] "RemoveContainer" containerID="44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c" Sep 15 21:54:48.688652 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:54:48.688626 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c\": container with ID starting with 44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c not found: ID does not exist" containerID="44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c" Sep 15 21:54:48.688750 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.688662 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c"} err="failed to get container status \"44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c\": rpc error: code = NotFound desc = could not find container \"44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c\": container with ID starting with 44069d7069f980cdc56af245519c2f113deb44b9900828eec9380c8e7d86928c not found: ID does not exist" Sep 15 21:54:48.689268 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.689245 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2"] Sep 15 21:54:48.691128 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.691091 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-2-w0-360e43-kserve-ff965fc99-k47n2"] Sep 15 21:54:48.700800 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.700779 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9"] Sep 15 21:54:48.702928 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.702908 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/premium-w0-360e43-kserve-7fdbbb6869-6kgk9"] Sep 15 21:54:48.712448 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.712426 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp"] Sep 15 21:54:48.715605 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:48.715587 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/distinct-w0-360e43-kserve-5f5f769fd8-hkphp"] Sep 15 21:54:49.148381 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:49.148352 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39bad9c6-6934-4522-b065-473d9891f551" path="/var/lib/kubelet/pods/39bad9c6-6934-4522-b065-473d9891f551/volumes" Sep 15 21:54:49.148741 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:49.148728 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c72862e-2fa9-463b-9d1b-c694e803c03e" path="/var/lib/kubelet/pods/3c72862e-2fa9-463b-9d1b-c694e803c03e/volumes" Sep 15 21:54:49.149066 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:49.149054 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" path="/var/lib/kubelet/pods/3fec278a-d859-42d9-bdf9-d9fdcb5e82d8/volumes" Sep 15 21:54:49.149698 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:54:49.149685 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7eede26-79bf-432b-b2fb-57973c63749f" path="/var/lib/kubelet/pods/d7eede26-79bf-432b-b2fb-57973c63749f/volumes" Sep 15 21:55:44.781616 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:55:44.781522 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q"] Sep 15 21:55:49.834633 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:55:49.834598 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z"] Sep 15 21:55:59.801453 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:55:59.801350 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" podUID="12ef3ce1-21fa-410e-b96a-ac087bb6aac5" containerName="main" containerID="cri-o://9cc9052a6fc97dd781fbed73e06465cbf380dfcadb44c214e4e01766a9ab06e3" gracePeriod=45 Sep 15 21:56:00.028509 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.028478 2575 generic.go:356] "Generic (PLEG): container finished" podID="12ef3ce1-21fa-410e-b96a-ac087bb6aac5" containerID="9cc9052a6fc97dd781fbed73e06465cbf380dfcadb44c214e4e01766a9ab06e3" exitCode=0 Sep 15 21:56:00.028663 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.028562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" event={"ID":"12ef3ce1-21fa-410e-b96a-ac087bb6aac5","Type":"ContainerDied","Data":"9cc9052a6fc97dd781fbed73e06465cbf380dfcadb44c214e4e01766a9ab06e3"} Sep 15 21:56:00.061847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.061787 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:56:00.068877 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.068850 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv"] Sep 15 21:56:00.069433 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069414 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa2c63c-388e-4045-bca3-e904dc2b90ff" containerName="maas-api" Sep 15 21:56:00.069486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069438 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa2c63c-388e-4045-bca3-e904dc2b90ff" containerName="maas-api" Sep 15 21:56:00.069486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069451 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c72862e-2fa9-463b-9d1b-c694e803c03e" containerName="main" Sep 15 21:56:00.069486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069459 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c72862e-2fa9-463b-9d1b-c694e803c03e" containerName="main" Sep 15 21:56:00.069486 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069481 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069489 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069498 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b85a0fe-10d7-441e-87de-dc46ba800b78" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069505 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b85a0fe-10d7-441e-87de-dc46ba800b78" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069516 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7eede26-79bf-432b-b2fb-57973c63749f" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069524 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7eede26-79bf-432b-b2fb-57973c63749f" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069543 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b991f35-71f9-4422-9896-068e62662be0" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069550 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b991f35-71f9-4422-9896-068e62662be0" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069561 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ef3ce1-21fa-410e-b96a-ac087bb6aac5" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069568 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ef3ce1-21fa-410e-b96a-ac087bb6aac5" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069587 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c09de5-9e48-4b1d-8810-de528c541431" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069594 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c09de5-9e48-4b1d-8810-de528c541431" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069600 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70761b49-8e24-48fc-a5f7-06a46b7da7bc" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069605 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70761b49-8e24-48fc-a5f7-06a46b7da7bc" containerName="maas-api" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069611 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bad9c6-6934-4522-b065-473d9891f551" containerName="main" Sep 15 21:56:00.069612 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069616 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bad9c6-6934-4522-b065-473d9891f551" containerName="main" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069685 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bad9c6-6934-4522-b065-473d9891f551" containerName="main" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069692 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c09de5-9e48-4b1d-8810-de528c541431" containerName="maas-api" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069697 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7eede26-79bf-432b-b2fb-57973c63749f" containerName="main" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069703 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b85a0fe-10d7-441e-87de-dc46ba800b78" containerName="maas-api" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ef3ce1-21fa-410e-b96a-ac087bb6aac5" containerName="main" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069715 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c72862e-2fa9-463b-9d1b-c694e803c03e" containerName="main" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069720 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fec278a-d859-42d9-bdf9-d9fdcb5e82d8" containerName="main" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069724 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b991f35-71f9-4422-9896-068e62662be0" containerName="maas-api" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069729 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70761b49-8e24-48fc-a5f7-06a46b7da7bc" containerName="maas-api" Sep 15 21:56:00.070039 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.069734 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa2c63c-388e-4045-bca3-e904dc2b90ff" containerName="maas-api" Sep 15 21:56:00.072484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.072468 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.076216 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.076195 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"sub-test-model-ce3ab0-kserve-self-signed-certs\"" Sep 15 21:56:00.090577 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.090554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv"] Sep 15 21:56:00.135584 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135552 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5snf\" (UniqueName: \"kubernetes.io/projected/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-kube-api-access-j5snf\") pod \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " Sep 15 21:56:00.135782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135608 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-dshm\") pod \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " Sep 15 21:56:00.135782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135640 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tmp-dir\") pod \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " Sep 15 21:56:00.135782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135663 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-home\") pod \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " Sep 15 21:56:00.135782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135693 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-model-cache\") pod \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " Sep 15 21:56:00.135782 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135726 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tls-certs\") pod \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\" (UID: \"12ef3ce1-21fa-410e-b96a-ac087bb6aac5\") " Sep 15 21:56:00.136040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-dshm\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.136040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3c87c-5e6c-4078-b606-7fb1245151c9-tls-certs\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.136040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-home\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.136040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135930 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12ef3ce1-21fa-410e-b96a-ac087bb6aac5" (UID: "12ef3ce1-21fa-410e-b96a-ac087bb6aac5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:00.136040 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.135987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqhp\" (UniqueName: \"kubernetes.io/projected/89a3c87c-5e6c-4078-b606-7fb1245151c9-kube-api-access-drqhp\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136060 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-model-cache" (OuterVolumeSpecName: "model-cache") pod "12ef3ce1-21fa-410e-b96a-ac087bb6aac5" (UID: "12ef3ce1-21fa-410e-b96a-ac087bb6aac5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-tmp-dir\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136162 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-home" (OuterVolumeSpecName: "home") pod "12ef3ce1-21fa-410e-b96a-ac087bb6aac5" (UID: "12ef3ce1-21fa-410e-b96a-ac087bb6aac5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-model-cache\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136240 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136252 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:00.136326 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.136261 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:00.138114 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.138067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12ef3ce1-21fa-410e-b96a-ac087bb6aac5" (UID: "12ef3ce1-21fa-410e-b96a-ac087bb6aac5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:56:00.138114 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.138072 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-dshm" (OuterVolumeSpecName: "dshm") pod "12ef3ce1-21fa-410e-b96a-ac087bb6aac5" (UID: "12ef3ce1-21fa-410e-b96a-ac087bb6aac5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:00.138298 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.138149 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-kube-api-access-j5snf" (OuterVolumeSpecName: "kube-api-access-j5snf") pod "12ef3ce1-21fa-410e-b96a-ac087bb6aac5" (UID: "12ef3ce1-21fa-410e-b96a-ac087bb6aac5"). InnerVolumeSpecName "kube-api-access-j5snf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:56:00.237216 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqhp\" (UniqueName: \"kubernetes.io/projected/89a3c87c-5e6c-4078-b606-7fb1245151c9-kube-api-access-drqhp\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237380 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-tmp-dir\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237380 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-model-cache\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237380 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-dshm\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237541 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3c87c-5e6c-4078-b606-7fb1245151c9-tls-certs\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237541 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-home\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237575 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:00.237613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237592 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:00.237613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237608 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5snf\" (UniqueName: \"kubernetes.io/projected/12ef3ce1-21fa-410e-b96a-ac087bb6aac5-kube-api-access-j5snf\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:00.237745 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-tmp-dir\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237836 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237816 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-home\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.237923 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.237902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-model-cache\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.239641 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.239623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-dshm\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.239947 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.239931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3c87c-5e6c-4078-b606-7fb1245151c9-tls-certs\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.246760 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.246734 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqhp\" (UniqueName: \"kubernetes.io/projected/89a3c87c-5e6c-4078-b606-7fb1245151c9-kube-api-access-drqhp\") pod \"sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.381890 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.381805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:00.659530 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.659506 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv"] Sep 15 21:56:00.661997 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:56:00.661973 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89a3c87c_5e6c_4078_b606_7fb1245151c9.slice/crio-619ed895a39953bd79312234dd578ce851a0718200724c4dc8237a5f77e04337 WatchSource:0}: Error finding container 619ed895a39953bd79312234dd578ce851a0718200724c4dc8237a5f77e04337: Status 404 returned error can't find the container with id 619ed895a39953bd79312234dd578ce851a0718200724c4dc8237a5f77e04337 Sep 15 21:56:00.775166 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.775087 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx"] Sep 15 21:56:00.775395 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:00.775365 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" podUID="72789745-49ef-426a-aa3e-f806c8420b99" containerName="maas-api" containerID="cri-o://5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697" gracePeriod=30 Sep 15 21:56:01.032323 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.032298 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:56:01.035093 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.035068 2575 generic.go:356] "Generic (PLEG): container finished" podID="72789745-49ef-426a-aa3e-f806c8420b99" containerID="5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697" exitCode=0 Sep 15 21:56:01.035268 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.035135 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" Sep 15 21:56:01.035268 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.035150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" event={"ID":"72789745-49ef-426a-aa3e-f806c8420b99","Type":"ContainerDied","Data":"5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697"} Sep 15 21:56:01.035268 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.035197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx" event={"ID":"72789745-49ef-426a-aa3e-f806c8420b99","Type":"ContainerDied","Data":"79bc3e7ae047c28f9c61a0a24612dc452ad13f9150916560e4bc6f62e441b276"} Sep 15 21:56:01.035268 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.035222 2575 scope.go:117] "RemoveContainer" containerID="5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697" Sep 15 21:56:01.036835 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.036803 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" event={"ID":"89a3c87c-5e6c-4078-b606-7fb1245151c9","Type":"ContainerStarted","Data":"04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197"} Sep 15 21:56:01.036958 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.036847 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" event={"ID":"89a3c87c-5e6c-4078-b606-7fb1245151c9","Type":"ContainerStarted","Data":"619ed895a39953bd79312234dd578ce851a0718200724c4dc8237a5f77e04337"} Sep 15 21:56:01.038221 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.038189 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" event={"ID":"12ef3ce1-21fa-410e-b96a-ac087bb6aac5","Type":"ContainerDied","Data":"d4e5c4dc86c8559d686797b0c91d65557bd9101e9e1f97a64930dd2793706c89"} Sep 15 21:56:01.038330 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.038224 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q" Sep 15 21:56:01.044939 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.044919 2575 scope.go:117] "RemoveContainer" containerID="5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697" Sep 15 21:56:01.045239 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:56:01.045217 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697\": container with ID starting with 5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697 not found: ID does not exist" containerID="5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697" Sep 15 21:56:01.045351 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.045247 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697"} err="failed to get container status \"5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697\": rpc error: code = NotFound desc = could not find container \"5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697\": container with ID starting with 5aa373067a4d2b6926b8e25d696c9708f5354dd7b7e9d1f39aee19e767fd0697 not found: ID does not exist" Sep 15 21:56:01.045351 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.045269 2575 scope.go:117] "RemoveContainer" containerID="9cc9052a6fc97dd781fbed73e06465cbf380dfcadb44c214e4e01766a9ab06e3" Sep 15 21:56:01.098469 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.098408 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" podStartSLOduration=1.098392114 podStartE2EDuration="1.098392114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:56:01.096813185 +0000 UTC m=+1810.436292630" watchObservedRunningTime="2026-09-15 21:56:01.098392114 +0000 UTC m=+1810.437871564" Sep 15 21:56:01.124182 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.124152 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q"] Sep 15 21:56:01.128045 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.128023 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/auth-test-model-ce3ab0-kserve-64667fc55-rqg2q"] Sep 15 21:56:01.146357 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.146338 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-maas-api-tls\") pod \"72789745-49ef-426a-aa3e-f806c8420b99\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " Sep 15 21:56:01.146480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.146363 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-metrics-certs\") pod \"72789745-49ef-426a-aa3e-f806c8420b99\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " Sep 15 21:56:01.146480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.146402 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72789745-49ef-426a-aa3e-f806c8420b99-openshift-service-ca\") pod \"72789745-49ef-426a-aa3e-f806c8420b99\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " Sep 15 21:56:01.146480 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.146419 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44pc8\" (UniqueName: \"kubernetes.io/projected/72789745-49ef-426a-aa3e-f806c8420b99-kube-api-access-44pc8\") pod \"72789745-49ef-426a-aa3e-f806c8420b99\" (UID: \"72789745-49ef-426a-aa3e-f806c8420b99\") " Sep 15 21:56:01.146821 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.146793 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72789745-49ef-426a-aa3e-f806c8420b99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72789745-49ef-426a-aa3e-f806c8420b99" (UID: "72789745-49ef-426a-aa3e-f806c8420b99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:56:01.148581 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.148559 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ef3ce1-21fa-410e-b96a-ac087bb6aac5" path="/var/lib/kubelet/pods/12ef3ce1-21fa-410e-b96a-ac087bb6aac5/volumes" Sep 15 21:56:01.148836 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.148812 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72789745-49ef-426a-aa3e-f806c8420b99" (UID: "72789745-49ef-426a-aa3e-f806c8420b99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:56:01.149081 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.149058 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72789745-49ef-426a-aa3e-f806c8420b99-kube-api-access-44pc8" (OuterVolumeSpecName: "kube-api-access-44pc8") pod "72789745-49ef-426a-aa3e-f806c8420b99" (UID: "72789745-49ef-426a-aa3e-f806c8420b99"). InnerVolumeSpecName "kube-api-access-44pc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:56:01.149219 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.149064 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72789745-49ef-426a-aa3e-f806c8420b99" (UID: "72789745-49ef-426a-aa3e-f806c8420b99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:56:01.247430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.247325 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:01.247430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.247384 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72789745-49ef-426a-aa3e-f806c8420b99-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:01.247430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.247396 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72789745-49ef-426a-aa3e-f806c8420b99-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:01.247430 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.247409 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pc8\" (UniqueName: \"kubernetes.io/projected/72789745-49ef-426a-aa3e-f806c8420b99-kube-api-access-44pc8\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:01.363553 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.363518 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx"] Sep 15 21:56:01.366661 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:01.366624 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-56e305-596d55f97-fgvcx"] Sep 15 21:56:03.146462 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:03.146428 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72789745-49ef-426a-aa3e-f806c8420b99" path="/var/lib/kubelet/pods/72789745-49ef-426a-aa3e-f806c8420b99/volumes" Sep 15 21:56:04.854426 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:04.854386 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" podUID="fa219405-3826-46fe-8c69-ea3042af755c" containerName="main" containerID="cri-o://34dd85b52836297574b2d2a3537c9f899c6a3baf616b889740e51e82c049404a" gracePeriod=45 Sep 15 21:56:05.062142 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.062085 2575 generic.go:356] "Generic (PLEG): container finished" podID="fa219405-3826-46fe-8c69-ea3042af755c" containerID="34dd85b52836297574b2d2a3537c9f899c6a3baf616b889740e51e82c049404a" exitCode=0 Sep 15 21:56:05.062334 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.062253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" event={"ID":"fa219405-3826-46fe-8c69-ea3042af755c","Type":"ContainerDied","Data":"34dd85b52836297574b2d2a3537c9f899c6a3baf616b889740e51e82c049404a"} Sep 15 21:56:05.109702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.109635 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:56:05.181905 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.181872 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz4xm\" (UniqueName: \"kubernetes.io/projected/fa219405-3826-46fe-8c69-ea3042af755c-kube-api-access-fz4xm\") pod \"fa219405-3826-46fe-8c69-ea3042af755c\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " Sep 15 21:56:05.182092 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.181961 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa219405-3826-46fe-8c69-ea3042af755c-tls-certs\") pod \"fa219405-3826-46fe-8c69-ea3042af755c\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " Sep 15 21:56:05.182092 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.181997 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-dshm\") pod \"fa219405-3826-46fe-8c69-ea3042af755c\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " Sep 15 21:56:05.182092 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.182022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-model-cache\") pod \"fa219405-3826-46fe-8c69-ea3042af755c\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " Sep 15 21:56:05.182092 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.182054 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-home\") pod \"fa219405-3826-46fe-8c69-ea3042af755c\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " Sep 15 21:56:05.182369 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.182121 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-tmp-dir\") pod \"fa219405-3826-46fe-8c69-ea3042af755c\" (UID: \"fa219405-3826-46fe-8c69-ea3042af755c\") " Sep 15 21:56:05.182603 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.182576 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa219405-3826-46fe-8c69-ea3042af755c" (UID: "fa219405-3826-46fe-8c69-ea3042af755c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:05.182712 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.182567 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa219405-3826-46fe-8c69-ea3042af755c" (UID: "fa219405-3826-46fe-8c69-ea3042af755c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:05.182788 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.182770 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-home" (OuterVolumeSpecName: "home") pod "fa219405-3826-46fe-8c69-ea3042af755c" (UID: "fa219405-3826-46fe-8c69-ea3042af755c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:05.185219 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.185183 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa219405-3826-46fe-8c69-ea3042af755c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa219405-3826-46fe-8c69-ea3042af755c" (UID: "fa219405-3826-46fe-8c69-ea3042af755c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:56:05.185337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.185310 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa219405-3826-46fe-8c69-ea3042af755c-kube-api-access-fz4xm" (OuterVolumeSpecName: "kube-api-access-fz4xm") pod "fa219405-3826-46fe-8c69-ea3042af755c" (UID: "fa219405-3826-46fe-8c69-ea3042af755c"). InnerVolumeSpecName "kube-api-access-fz4xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:56:05.185404 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.185354 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-dshm" (OuterVolumeSpecName: "dshm") pod "fa219405-3826-46fe-8c69-ea3042af755c" (UID: "fa219405-3826-46fe-8c69-ea3042af755c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:56:05.283578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.283544 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa219405-3826-46fe-8c69-ea3042af755c-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:05.283578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.283581 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:05.283578 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.283594 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:05.283819 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.283605 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:05.283819 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.283617 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa219405-3826-46fe-8c69-ea3042af755c-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:05.283819 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:05.283629 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4xm\" (UniqueName: \"kubernetes.io/projected/fa219405-3826-46fe-8c69-ea3042af755c-kube-api-access-fz4xm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:56:06.068466 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:06.068438 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" Sep 15 21:56:06.068864 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:06.068440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z" event={"ID":"fa219405-3826-46fe-8c69-ea3042af755c","Type":"ContainerDied","Data":"d44a4a608d3ced1cc2f04b6b0bdf885def40e3daf29d2b2ff1d4f0a008d99c20"} Sep 15 21:56:06.068864 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:06.068564 2575 scope.go:117] "RemoveContainer" containerID="34dd85b52836297574b2d2a3537c9f899c6a3baf616b889740e51e82c049404a" Sep 15 21:56:06.133654 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:06.133625 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z"] Sep 15 21:56:06.146248 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:06.146224 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/premium-w4-56e305-kserve-78dc857c76-cth5z"] Sep 15 21:56:07.148556 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:07.148522 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa219405-3826-46fe-8c69-ea3042af755c" path="/var/lib/kubelet/pods/fa219405-3826-46fe-8c69-ea3042af755c/volumes" Sep 15 21:56:10.382337 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:10.382299 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:10.382768 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:10.382652 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:10.395514 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:10.395491 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:56:11.106651 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:56:11.106620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:57:37.519702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:37.519671 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv"] Sep 15 21:57:52.538702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.538640 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" podUID="89a3c87c-5e6c-4078-b606-7fb1245151c9" containerName="main" containerID="cri-o://04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197" gracePeriod=45 Sep 15 21:57:52.803933 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.803906 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:57:52.838709 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.838678 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drqhp\" (UniqueName: \"kubernetes.io/projected/89a3c87c-5e6c-4078-b606-7fb1245151c9-kube-api-access-drqhp\") pod \"89a3c87c-5e6c-4078-b606-7fb1245151c9\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " Sep 15 21:57:52.838893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.838783 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-dshm\") pod \"89a3c87c-5e6c-4078-b606-7fb1245151c9\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " Sep 15 21:57:52.838893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.838810 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3c87c-5e6c-4078-b606-7fb1245151c9-tls-certs\") pod \"89a3c87c-5e6c-4078-b606-7fb1245151c9\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " Sep 15 21:57:52.838893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.838851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-home\") pod \"89a3c87c-5e6c-4078-b606-7fb1245151c9\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " Sep 15 21:57:52.838893 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.838876 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-tmp-dir\") pod \"89a3c87c-5e6c-4078-b606-7fb1245151c9\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " Sep 15 21:57:52.839179 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.838899 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-model-cache\") pod \"89a3c87c-5e6c-4078-b606-7fb1245151c9\" (UID: \"89a3c87c-5e6c-4078-b606-7fb1245151c9\") " Sep 15 21:57:52.839308 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.839282 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "89a3c87c-5e6c-4078-b606-7fb1245151c9" (UID: "89a3c87c-5e6c-4078-b606-7fb1245151c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:52.839308 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.839293 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-home" (OuterVolumeSpecName: "home") pod "89a3c87c-5e6c-4078-b606-7fb1245151c9" (UID: "89a3c87c-5e6c-4078-b606-7fb1245151c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:52.839443 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.839237 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89a3c87c-5e6c-4078-b606-7fb1245151c9" (UID: "89a3c87c-5e6c-4078-b606-7fb1245151c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:52.841176 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.841148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a3c87c-5e6c-4078-b606-7fb1245151c9-kube-api-access-drqhp" (OuterVolumeSpecName: "kube-api-access-drqhp") pod "89a3c87c-5e6c-4078-b606-7fb1245151c9" (UID: "89a3c87c-5e6c-4078-b606-7fb1245151c9"). InnerVolumeSpecName "kube-api-access-drqhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:57:52.841296 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.841180 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-dshm" (OuterVolumeSpecName: "dshm") pod "89a3c87c-5e6c-4078-b606-7fb1245151c9" (UID: "89a3c87c-5e6c-4078-b606-7fb1245151c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:52.841533 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.841511 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a3c87c-5e6c-4078-b606-7fb1245151c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89a3c87c-5e6c-4078-b606-7fb1245151c9" (UID: "89a3c87c-5e6c-4078-b606-7fb1245151c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:57:52.939922 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.939890 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:57:52.939922 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.939918 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3c87c-5e6c-4078-b606-7fb1245151c9-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:57:52.939922 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.939928 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:57:52.940185 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.939936 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:57:52.940185 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.939944 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3c87c-5e6c-4078-b606-7fb1245151c9-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:57:52.940185 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:52.939952 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drqhp\" (UniqueName: \"kubernetes.io/projected/89a3c87c-5e6c-4078-b606-7fb1245151c9-kube-api-access-drqhp\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:57:53.597220 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.597176 2575 generic.go:356] "Generic (PLEG): container finished" podID="89a3c87c-5e6c-4078-b606-7fb1245151c9" containerID="04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197" exitCode=0 Sep 15 21:57:53.597220 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.597220 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" Sep 15 21:57:53.597685 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.597262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" event={"ID":"89a3c87c-5e6c-4078-b606-7fb1245151c9","Type":"ContainerDied","Data":"04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197"} Sep 15 21:57:53.597685 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.597309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv" event={"ID":"89a3c87c-5e6c-4078-b606-7fb1245151c9","Type":"ContainerDied","Data":"619ed895a39953bd79312234dd578ce851a0718200724c4dc8237a5f77e04337"} Sep 15 21:57:53.597685 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.597330 2575 scope.go:117] "RemoveContainer" containerID="04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197" Sep 15 21:57:53.606130 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.606112 2575 scope.go:117] "RemoveContainer" containerID="04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197" Sep 15 21:57:53.606426 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:57:53.606407 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197\": container with ID starting with 04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197 not found: ID does not exist" containerID="04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197" Sep 15 21:57:53.606492 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.606436 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197"} err="failed to get container status \"04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197\": rpc error: code = NotFound desc = could not find container \"04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197\": container with ID starting with 04c165d5788f21ab1450629ecfa99d1d0c00522282f10332fbfff6d5c7f8c197 not found: ID does not exist" Sep 15 21:57:53.617847 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.617824 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv"] Sep 15 21:57:53.622650 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:53.622630 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/sub-test-model-ce3ab0-kserve-7c56d6f9bc-x6mpv"] Sep 15 21:57:55.147549 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:57:55.147516 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a3c87c-5e6c-4078-b606-7fb1245151c9" path="/var/lib/kubelet/pods/89a3c87c-5e6c-4078-b606-7fb1245151c9/volumes" Sep 15 21:58:07.940166 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940134 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj"] Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940491 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72789745-49ef-426a-aa3e-f806c8420b99" containerName="maas-api" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940504 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="72789745-49ef-426a-aa3e-f806c8420b99" containerName="maas-api" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940526 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a3c87c-5e6c-4078-b606-7fb1245151c9" containerName="main" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940532 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a3c87c-5e6c-4078-b606-7fb1245151c9" containerName="main" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940542 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa219405-3826-46fe-8c69-ea3042af755c" containerName="main" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940547 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa219405-3826-46fe-8c69-ea3042af755c" containerName="main" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940602 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="72789745-49ef-426a-aa3e-f806c8420b99" containerName="maas-api" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940610 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa219405-3826-46fe-8c69-ea3042af755c" containerName="main" Sep 15 21:58:07.940613 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.940616 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a3c87c-5e6c-4078-b606-7fb1245151c9" containerName="main" Sep 15 21:58:07.945175 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.945158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:07.948053 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.948031 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"openshift-service-ca.crt\"" Sep 15 21:58:07.948235 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.948066 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"kube-root-ca.crt\"" Sep 15 21:58:07.949440 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.949417 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"default-dockercfg-vxmvf\"" Sep 15 21:58:07.949440 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.949432 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"rate-test-model-e7a648-kserve-self-signed-certs\"" Sep 15 21:58:07.957028 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:07.957006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj"] Sep 15 21:58:08.065692 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.065663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tmp-dir\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.065865 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.065701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-model-cache\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.065865 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.065722 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvp7\" (UniqueName: \"kubernetes.io/projected/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-kube-api-access-6zvp7\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.065865 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.065770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-home\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.065865 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.065826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-dshm\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.065865 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.065858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tls-certs\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166450 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tmp-dir\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166450 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-model-cache\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvp7\" (UniqueName: \"kubernetes.io/projected/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-kube-api-access-6zvp7\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-home\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-dshm\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166702 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tls-certs\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166917 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tmp-dir\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166964 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-model-cache\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.166964 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.166943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-home\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.169011 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.168985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-dshm\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.169182 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.169165 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tls-certs\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.176962 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.176937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvp7\" (UniqueName: \"kubernetes.io/projected/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-kube-api-access-6zvp7\") pod \"rate-test-model-e7a648-kserve-875cbdcf9-spxqj\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.254618 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.254553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:08.535700 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.535670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj"] Sep 15 21:58:08.537287 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:58:08.537249 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf2002c1_cc3c_4b93_85ec_58ef0d4e18c4.slice/crio-919d0fb0be001ac95decb0d59896091de971b83fe953d6fffd6a6f2bdda33342 WatchSource:0}: Error finding container 919d0fb0be001ac95decb0d59896091de971b83fe953d6fffd6a6f2bdda33342: Status 404 returned error can't find the container with id 919d0fb0be001ac95decb0d59896091de971b83fe953d6fffd6a6f2bdda33342 Sep 15 21:58:08.674376 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.674341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" event={"ID":"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4","Type":"ContainerStarted","Data":"7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066"} Sep 15 21:58:08.674517 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.674391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" event={"ID":"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4","Type":"ContainerStarted","Data":"919d0fb0be001ac95decb0d59896091de971b83fe953d6fffd6a6f2bdda33342"} Sep 15 21:58:08.699549 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:08.699500 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" podStartSLOduration=1.6994839069999998 podStartE2EDuration="1.699483907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:58:08.698782465 +0000 UTC m=+1938.038261915" watchObservedRunningTime="2026-09-15 21:58:08.699483907 +0000 UTC m=+1938.038963357" Sep 15 21:58:18.255093 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:18.255053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:18.255521 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:18.255143 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:18.267715 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:18.267687 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:18.733436 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:18.733410 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:58:53.127968 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:53.127937 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj"] Sep 15 21:58:59.477293 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.477263 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb"] Sep 15 21:58:59.480943 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.480921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.484247 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.484225 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0886df-serving-cert\"" Sep 15 21:58:59.503389 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.503354 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb"] Sep 15 21:58:59.508519 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.508481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.508645 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.508544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-metrics-certs\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.508645 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.508573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.508645 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.508621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkzj\" (UniqueName: \"kubernetes.io/projected/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-kube-api-access-xwkzj\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.609318 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.609283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.609496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.609330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkzj\" (UniqueName: \"kubernetes.io/projected/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-kube-api-access-xwkzj\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.609496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.609394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.609496 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.609421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-metrics-certs\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.609692 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:58:59.609518 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-serving-cert: secret "maas-api-e2e-ipp-a-0886df-serving-cert" not found Sep 15 21:58:59.609692 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:58:59.609587 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls podName:d0bb5770-6b6b-4581-8a8b-15fa5e1ad395 nodeName:}" failed. No retries permitted until 2026-09-15 21:59:00.109567947 +0000 UTC m=+1989.449047375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls") pod "maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" (UID: "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395") : secret "maas-api-e2e-ipp-a-0886df-serving-cert" not found Sep 15 21:58:59.610027 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.610000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.611900 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.611869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-metrics-certs\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:58:59.623896 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:58:59.623868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkzj\" (UniqueName: \"kubernetes.io/projected/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-kube-api-access-xwkzj\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:59:00.114477 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:00.114442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:59:00.117199 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:00.117161 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:59:00.394377 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:00.394283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:59:00.672537 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:00.672508 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb"] Sep 15 21:59:00.673753 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:59:00.673728 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bb5770_6b6b_4581_8a8b_15fa5e1ad395.slice/crio-235c79b3b74ee0f8cc83e0033e03d0709b87afa0f166c93e671afdf7d8204e5f WatchSource:0}: Error finding container 235c79b3b74ee0f8cc83e0033e03d0709b87afa0f166c93e671afdf7d8204e5f: Status 404 returned error can't find the container with id 235c79b3b74ee0f8cc83e0033e03d0709b87afa0f166c93e671afdf7d8204e5f Sep 15 21:59:00.675530 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:00.675506 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:59:00.925010 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:00.924920 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" event={"ID":"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395","Type":"ContainerStarted","Data":"235c79b3b74ee0f8cc83e0033e03d0709b87afa0f166c93e671afdf7d8204e5f"} Sep 15 21:59:01.931329 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:01.931295 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" event={"ID":"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395","Type":"ContainerStarted","Data":"3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e"} Sep 15 21:59:01.931807 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:01.931364 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:59:01.952663 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:01.952611 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" podStartSLOduration=2.417434201 podStartE2EDuration="2.952592175s" totalImagesPullingTime="535.157974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:01.951359894 +0000 UTC m=+1991.290839345" watchObservedRunningTime="2026-09-15 21:59:01.952592175 +0000 UTC m=+1991.292071628" Sep 15 21:59:07.943826 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:07.943799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 21:59:08.147484 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.147421 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" podUID="bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" containerName="main" containerID="cri-o://7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066" gracePeriod=45 Sep 15 21:59:08.403279 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.403251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:59:08.495422 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-dshm\") pod \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " Sep 15 21:59:08.495592 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tls-certs\") pod \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " Sep 15 21:59:08.495592 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-model-cache\") pod \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " Sep 15 21:59:08.495592 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495500 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-home\") pod \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " Sep 15 21:59:08.495592 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495559 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tmp-dir\") pod \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " Sep 15 21:59:08.495826 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495604 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zvp7\" (UniqueName: \"kubernetes.io/projected/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-kube-api-access-6zvp7\") pod \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\" (UID: \"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4\") " Sep 15 21:59:08.495826 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495749 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" (UID: "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:59:08.495920 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495801 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-home" (OuterVolumeSpecName: "home") pod "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" (UID: "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:59:08.495920 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495848 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:59:08.495920 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.495876 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" (UID: "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:59:08.497768 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.497747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-dshm" (OuterVolumeSpecName: "dshm") pod "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" (UID: "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:59:08.497866 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.497840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-kube-api-access-6zvp7" (OuterVolumeSpecName: "kube-api-access-6zvp7") pod "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" (UID: "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4"). InnerVolumeSpecName "kube-api-access-6zvp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:59:08.497910 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.497887 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" (UID: "bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:59:08.597067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.596989 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvp7\" (UniqueName: \"kubernetes.io/projected/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-kube-api-access-6zvp7\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:59:08.597067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.597016 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:59:08.597067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.597028 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:59:08.597067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.597037 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:59:08.597067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.597044 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 21:59:08.967280 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.967248 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" containerID="7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066" exitCode=0 Sep 15 21:59:08.967680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.967293 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" Sep 15 21:59:08.967680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.967338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" event={"ID":"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4","Type":"ContainerDied","Data":"7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066"} Sep 15 21:59:08.967680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.967383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj" event={"ID":"bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4","Type":"ContainerDied","Data":"919d0fb0be001ac95decb0d59896091de971b83fe953d6fffd6a6f2bdda33342"} Sep 15 21:59:08.967680 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.967403 2575 scope.go:117] "RemoveContainer" containerID="7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066" Sep 15 21:59:08.977155 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.977138 2575 scope.go:117] "RemoveContainer" containerID="7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066" Sep 15 21:59:08.977453 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:59:08.977433 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066\": container with ID starting with 7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066 not found: ID does not exist" containerID="7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066" Sep 15 21:59:08.977503 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.977462 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066"} err="failed to get container status \"7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066\": rpc error: code = NotFound desc = could not find container \"7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066\": container with ID starting with 7feaf674f1f4c6dde3a000abb1b9c3993f167875b1cfb61531cea931edd03066 not found: ID does not exist" Sep 15 21:59:08.993067 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.993041 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj"] Sep 15 21:59:08.999846 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:08.999826 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/rate-test-model-e7a648-kserve-875cbdcf9-spxqj"] Sep 15 21:59:09.151721 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:09.151679 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" path="/var/lib/kubelet/pods/bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4/volumes" Sep 15 21:59:34.604441 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.604402 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r"] Sep 15 21:59:34.605028 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.605010 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" containerName="main" Sep 15 21:59:34.605028 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.605030 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" containerName="main" Sep 15 21:59:34.605210 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.605132 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2002c1-cc3c-4b93-85ec-58ef0d4e18c4" containerName="main" Sep 15 21:59:34.608342 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.608316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.611513 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.611492 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1297a3-serving-cert\"" Sep 15 21:59:34.621189 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.621156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41390e4-4285-421e-97d8-12d348c1d41f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.621315 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.621288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-metrics-certs\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.621389 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.621369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6pj8\" (UniqueName: \"kubernetes.io/projected/b41390e4-4285-421e-97d8-12d348c1d41f-kube-api-access-k6pj8\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.621444 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.621420 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.709924 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.709889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r"] Sep 15 21:59:34.722502 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.722474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6pj8\" (UniqueName: \"kubernetes.io/projected/b41390e4-4285-421e-97d8-12d348c1d41f-kube-api-access-k6pj8\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.722679 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.722512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.722679 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:59:34.722631 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-serving-cert: secret "maas-api-e2e-ipp-b-1297a3-serving-cert" not found Sep 15 21:59:34.722679 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.722634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41390e4-4285-421e-97d8-12d348c1d41f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.722872 ip-10-0-130-52 kubenswrapper[2575]: E0915 21:59:34.722703 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls podName:b41390e4-4285-421e-97d8-12d348c1d41f nodeName:}" failed. No retries permitted until 2026-09-15 21:59:35.222681471 +0000 UTC m=+2024.562160901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls") pod "maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" (UID: "b41390e4-4285-421e-97d8-12d348c1d41f") : secret "maas-api-e2e-ipp-b-1297a3-serving-cert" not found Sep 15 21:59:34.722872 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.722805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-metrics-certs\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.723446 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.723422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41390e4-4285-421e-97d8-12d348c1d41f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.725730 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.725707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-metrics-certs\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:34.743033 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:34.743001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6pj8\" (UniqueName: \"kubernetes.io/projected/b41390e4-4285-421e-97d8-12d348c1d41f-kube-api-access-k6pj8\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:35.226282 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:35.226247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:35.228773 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:35.228755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:35.521671 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:35.521586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:35.807793 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:35.807637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r"] Sep 15 21:59:35.810784 ip-10-0-130-52 kubenswrapper[2575]: W0915 21:59:35.810754 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41390e4_4285_421e_97d8_12d348c1d41f.slice/crio-387a0e640291d8001cc2de6e647a06c61e0c7bd0b4a2387d29d38a3c105d4d8d WatchSource:0}: Error finding container 387a0e640291d8001cc2de6e647a06c61e0c7bd0b4a2387d29d38a3c105d4d8d: Status 404 returned error can't find the container with id 387a0e640291d8001cc2de6e647a06c61e0c7bd0b4a2387d29d38a3c105d4d8d Sep 15 21:59:36.112789 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:36.112701 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" event={"ID":"b41390e4-4285-421e-97d8-12d348c1d41f","Type":"ContainerStarted","Data":"387a0e640291d8001cc2de6e647a06c61e0c7bd0b4a2387d29d38a3c105d4d8d"} Sep 15 21:59:37.119508 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:37.119471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" event={"ID":"b41390e4-4285-421e-97d8-12d348c1d41f","Type":"ContainerStarted","Data":"acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a"} Sep 15 21:59:37.119937 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:37.119573 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 21:59:37.151529 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:37.151400 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" podStartSLOduration=2.792643317 podStartE2EDuration="3.151378168s" totalImagesPullingTime="358.734851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:37.149504988 +0000 UTC m=+2026.488984439" watchObservedRunningTime="2026-09-15 21:59:37.151378168 +0000 UTC m=+2026.490857620" Sep 15 21:59:43.130965 ip-10-0-130-52 kubenswrapper[2575]: I0915 21:59:43.130937 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 22:00:10.954613 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.954580 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m"] Sep 15 22:00:10.958457 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.958434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:10.963576 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.963554 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0886df\"/\"ipp-route-0886df-kserve-self-signed-certs\"" Sep 15 22:00:10.963734 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.963552 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0886df\"/\"openshift-service-ca.crt\"" Sep 15 22:00:10.963734 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.963563 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0886df\"/\"kube-root-ca.crt\"" Sep 15 22:00:10.963734 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.963613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0886df\"/\"default-dockercfg-2blk8\"" Sep 15 22:00:10.978424 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:10.978395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m"] Sep 15 22:00:11.063175 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.063141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-model-cache\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.063392 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.063206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tls-certs\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.063392 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.063275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tmp-dir\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.063392 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.063320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7rt\" (UniqueName: \"kubernetes.io/projected/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-kube-api-access-vc7rt\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.063392 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.063357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-home\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.063392 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.063387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-dshm\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.164669 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.164627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-home\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.164669 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.164670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-dshm\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.164947 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.164810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-model-cache\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.164947 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.164875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tls-certs\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.164947 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.164917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tmp-dir\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.164947 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.164937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7rt\" (UniqueName: \"kubernetes.io/projected/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-kube-api-access-vc7rt\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.165212 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.165145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-home\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.165260 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.165234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-model-cache\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.165335 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.165316 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tmp-dir\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.167146 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.167127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-dshm\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.167632 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.167609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tls-certs\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.175717 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.175690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7rt\" (UniqueName: \"kubernetes.io/projected/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-kube-api-access-vc7rt\") pod \"ipp-route-0886df-kserve-59b976464c-8vb8m\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.268817 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.268721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:11.560214 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:00:11.560178 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9a7e7b2_36ea_4293_ac86_4ac1bd780b12.slice/crio-7b20b0da00286a583359b1239636692eab73b91b193ea5826d0a0f2b474f673b WatchSource:0}: Error finding container 7b20b0da00286a583359b1239636692eab73b91b193ea5826d0a0f2b474f673b: Status 404 returned error can't find the container with id 7b20b0da00286a583359b1239636692eab73b91b193ea5826d0a0f2b474f673b Sep 15 22:00:11.564579 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:11.564550 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m"] Sep 15 22:00:12.287953 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:12.287917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" event={"ID":"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12","Type":"ContainerStarted","Data":"ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7"} Sep 15 22:00:12.288373 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:12.287966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" event={"ID":"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12","Type":"ContainerStarted","Data":"7b20b0da00286a583359b1239636692eab73b91b193ea5826d0a0f2b474f673b"} Sep 15 22:00:12.314912 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:12.314859 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" podStartSLOduration=2.314840774 podStartE2EDuration="2.314840774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:00:12.31229894 +0000 UTC m=+2061.651778390" watchObservedRunningTime="2026-09-15 22:00:12.314840774 +0000 UTC m=+2061.654320290" Sep 15 22:00:21.269848 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:21.269813 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:21.269848 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:21.269860 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:21.283281 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:21.283244 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:21.345045 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:21.345016 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:00:46.448690 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.448656 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln"] Sep 15 22:00:46.452472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.452450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.455838 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.455814 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-3596d6-serving-cert\"" Sep 15 22:00:46.462306 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.462281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln"] Sep 15 22:00:46.595036 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.595002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.595036 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.595041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rws\" (UniqueName: \"kubernetes.io/projected/bf82155e-fad3-47e0-a65a-c94b0195d2f8-kube-api-access-c4rws\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.595316 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.595156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf82155e-fad3-47e0-a65a-c94b0195d2f8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.595316 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.595211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.695646 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.695618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf82155e-fad3-47e0-a65a-c94b0195d2f8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.695832 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.695656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.695832 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.695700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.695832 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.695718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rws\" (UniqueName: \"kubernetes.io/projected/bf82155e-fad3-47e0-a65a-c94b0195d2f8-kube-api-access-c4rws\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.696006 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:00:46.695838 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-serving-cert: secret "maas-api-e2e-ipp-cleanup-3596d6-serving-cert" not found Sep 15 22:00:46.696006 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:00:46.695888 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls podName:bf82155e-fad3-47e0-a65a-c94b0195d2f8 nodeName:}" failed. No retries permitted until 2026-09-15 22:00:47.19587393 +0000 UTC m=+2096.535353359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" (UID: "bf82155e-fad3-47e0-a65a-c94b0195d2f8") : secret "maas-api-e2e-ipp-cleanup-3596d6-serving-cert" not found Sep 15 22:00:46.696362 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.696342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf82155e-fad3-47e0-a65a-c94b0195d2f8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.698378 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.698355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:46.707323 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:46.707255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rws\" (UniqueName: \"kubernetes.io/projected/bf82155e-fad3-47e0-a65a-c94b0195d2f8-kube-api-access-c4rws\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:47.200511 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:47.200477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:47.200703 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:00:47.200628 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-serving-cert: secret "maas-api-e2e-ipp-cleanup-3596d6-serving-cert" not found Sep 15 22:00:47.200780 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:00:47.200705 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls podName:bf82155e-fad3-47e0-a65a-c94b0195d2f8 nodeName:}" failed. No retries permitted until 2026-09-15 22:00:48.20068669 +0000 UTC m=+2097.540166133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" (UID: "bf82155e-fad3-47e0-a65a-c94b0195d2f8") : secret "maas-api-e2e-ipp-cleanup-3596d6-serving-cert" not found Sep 15 22:00:48.209876 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:48.209834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:48.212541 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:48.212518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:48.265590 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:48.265558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:48.552262 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:48.552174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln"] Sep 15 22:00:48.555148 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:00:48.555120 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf82155e_fad3_47e0_a65a_c94b0195d2f8.slice/crio-e2e046b62296cc0b725ca291301208ac0fbecaef39af70fcdaf07b11773a118b WatchSource:0}: Error finding container e2e046b62296cc0b725ca291301208ac0fbecaef39af70fcdaf07b11773a118b: Status 404 returned error can't find the container with id e2e046b62296cc0b725ca291301208ac0fbecaef39af70fcdaf07b11773a118b Sep 15 22:00:49.474541 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:49.474503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" event={"ID":"bf82155e-fad3-47e0-a65a-c94b0195d2f8","Type":"ContainerStarted","Data":"f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c"} Sep 15 22:00:49.474915 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:49.474571 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:00:49.474915 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:49.474592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" event={"ID":"bf82155e-fad3-47e0-a65a-c94b0195d2f8","Type":"ContainerStarted","Data":"e2e046b62296cc0b725ca291301208ac0fbecaef39af70fcdaf07b11773a118b"} Sep 15 22:00:49.494920 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:49.494872 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" podStartSLOduration=3.005522774 podStartE2EDuration="3.494854852s" totalImagesPullingTime="489.332078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:00:49.494410011 +0000 UTC m=+2098.833889463" watchObservedRunningTime="2026-09-15 22:00:49.494854852 +0000 UTC m=+2098.834334337" Sep 15 22:00:55.486018 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:00:55.485988 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:01:23.986725 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:23.986681 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln"] Sep 15 22:01:23.987299 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:23.986945 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" podUID="bf82155e-fad3-47e0-a65a-c94b0195d2f8" containerName="maas-api" containerID="cri-o://f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c" gracePeriod=30 Sep 15 22:01:24.244751 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.244693 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:01:24.325651 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.325619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-metrics-certs\") pod \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " Sep 15 22:01:24.325817 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.325669 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") pod \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " Sep 15 22:01:24.325817 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.325752 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4rws\" (UniqueName: \"kubernetes.io/projected/bf82155e-fad3-47e0-a65a-c94b0195d2f8-kube-api-access-c4rws\") pod \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " Sep 15 22:01:24.325817 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.325803 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf82155e-fad3-47e0-a65a-c94b0195d2f8-openshift-service-ca\") pod \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\" (UID: \"bf82155e-fad3-47e0-a65a-c94b0195d2f8\") " Sep 15 22:01:24.326301 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.326257 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf82155e-fad3-47e0-a65a-c94b0195d2f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf82155e-fad3-47e0-a65a-c94b0195d2f8" (UID: "bf82155e-fad3-47e0-a65a-c94b0195d2f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 22:01:24.328149 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.328116 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82155e-fad3-47e0-a65a-c94b0195d2f8-kube-api-access-c4rws" (OuterVolumeSpecName: "kube-api-access-c4rws") pod "bf82155e-fad3-47e0-a65a-c94b0195d2f8" (UID: "bf82155e-fad3-47e0-a65a-c94b0195d2f8"). InnerVolumeSpecName "kube-api-access-c4rws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:01:24.328422 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.328400 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf82155e-fad3-47e0-a65a-c94b0195d2f8" (UID: "bf82155e-fad3-47e0-a65a-c94b0195d2f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:01:24.328524 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.328427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf82155e-fad3-47e0-a65a-c94b0195d2f8" (UID: "bf82155e-fad3-47e0-a65a-c94b0195d2f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:01:24.427388 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.427354 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4rws\" (UniqueName: \"kubernetes.io/projected/bf82155e-fad3-47e0-a65a-c94b0195d2f8-kube-api-access-c4rws\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:24.427388 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.427391 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf82155e-fad3-47e0-a65a-c94b0195d2f8-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:24.427580 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.427407 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:24.427580 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.427421 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf82155e-fad3-47e0-a65a-c94b0195d2f8-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:24.655310 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.655221 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf82155e-fad3-47e0-a65a-c94b0195d2f8" containerID="f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c" exitCode=0 Sep 15 22:01:24.655310 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.655263 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" Sep 15 22:01:24.655310 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.655294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" event={"ID":"bf82155e-fad3-47e0-a65a-c94b0195d2f8","Type":"ContainerDied","Data":"f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c"} Sep 15 22:01:24.655550 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.655344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln" event={"ID":"bf82155e-fad3-47e0-a65a-c94b0195d2f8","Type":"ContainerDied","Data":"e2e046b62296cc0b725ca291301208ac0fbecaef39af70fcdaf07b11773a118b"} Sep 15 22:01:24.655550 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.655366 2575 scope.go:117] "RemoveContainer" containerID="f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c" Sep 15 22:01:24.666036 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.666013 2575 scope.go:117] "RemoveContainer" containerID="f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c" Sep 15 22:01:24.666410 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:01:24.666385 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c\": container with ID starting with f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c not found: ID does not exist" containerID="f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c" Sep 15 22:01:24.666515 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.666421 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c"} err="failed to get container status \"f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c\": rpc error: code = NotFound desc = could not find container \"f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c\": container with ID starting with f4461ddf4c8efef51fe1a3e84640c6f097ae5644392f2852407f479e69cfa97c not found: ID does not exist" Sep 15 22:01:24.687530 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.687505 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln"] Sep 15 22:01:24.702138 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:24.702115 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3596d6-f6f5b9bc8-4qmln"] Sep 15 22:01:25.147275 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:25.147230 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf82155e-fad3-47e0-a65a-c94b0195d2f8" path="/var/lib/kubelet/pods/bf82155e-fad3-47e0-a65a-c94b0195d2f8/volumes" Sep 15 22:01:53.385250 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.385211 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb"] Sep 15 22:01:53.385731 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.385557 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" podUID="d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" containerName="maas-api" containerID="cri-o://3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e" gracePeriod=30 Sep 15 22:01:53.644702 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.644644 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 22:01:53.693859 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.693825 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-openshift-service-ca\") pod \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " Sep 15 22:01:53.694044 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.693872 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwkzj\" (UniqueName: \"kubernetes.io/projected/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-kube-api-access-xwkzj\") pod \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " Sep 15 22:01:53.694044 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.693941 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls\") pod \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " Sep 15 22:01:53.694044 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.693970 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-metrics-certs\") pod \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\" (UID: \"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395\") " Sep 15 22:01:53.694353 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.694264 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" (UID: "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 22:01:53.696933 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.696901 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" (UID: "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:01:53.697251 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.697223 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" (UID: "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:01:53.697347 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.697259 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-kube-api-access-xwkzj" (OuterVolumeSpecName: "kube-api-access-xwkzj") pod "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" (UID: "d0bb5770-6b6b-4581-8a8b-15fa5e1ad395"). InnerVolumeSpecName "kube-api-access-xwkzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:01:53.795454 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.795424 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:53.795454 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.795450 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwkzj\" (UniqueName: \"kubernetes.io/projected/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-kube-api-access-xwkzj\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:53.795656 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.795462 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:53.795656 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.795476 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:01:53.803395 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.803369 2575 generic.go:356] "Generic (PLEG): container finished" podID="d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" containerID="3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e" exitCode=0 Sep 15 22:01:53.803542 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.803420 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" Sep 15 22:01:53.803542 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.803455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" event={"ID":"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395","Type":"ContainerDied","Data":"3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e"} Sep 15 22:01:53.803542 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.803520 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb" event={"ID":"d0bb5770-6b6b-4581-8a8b-15fa5e1ad395","Type":"ContainerDied","Data":"235c79b3b74ee0f8cc83e0033e03d0709b87afa0f166c93e671afdf7d8204e5f"} Sep 15 22:01:53.803710 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.803548 2575 scope.go:117] "RemoveContainer" containerID="3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e" Sep 15 22:01:53.814153 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.814137 2575 scope.go:117] "RemoveContainer" containerID="3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e" Sep 15 22:01:53.814509 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:01:53.814483 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e\": container with ID starting with 3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e not found: ID does not exist" containerID="3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e" Sep 15 22:01:53.814590 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.814521 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e"} err="failed to get container status \"3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e\": rpc error: code = NotFound desc = could not find container \"3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e\": container with ID starting with 3fcb1a198e489abbe36587eab9e0c49abecd9a72260bff1b8640f37dcc902c1e not found: ID does not exist" Sep 15 22:01:53.834808 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.834785 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb"] Sep 15 22:01:53.837399 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:53.837372 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0886df-7b9d9c9b9d-pvvpb"] Sep 15 22:01:55.147570 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:55.147535 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" path="/var/lib/kubelet/pods/d0bb5770-6b6b-4581-8a8b-15fa5e1ad395/volumes" Sep 15 22:01:59.172955 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:01:59.172920 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m"] Sep 15 22:02:14.193160 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.193113 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" podUID="c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" containerName="main" containerID="cri-o://ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7" gracePeriod=45 Sep 15 22:02:14.439911 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.439888 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:02:14.469819 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.469738 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tls-certs\") pod \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " Sep 15 22:02:14.469819 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.469777 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-dshm\") pod \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " Sep 15 22:02:14.469819 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.469811 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-home\") pod \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " Sep 15 22:02:14.470156 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.469901 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tmp-dir\") pod \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " Sep 15 22:02:14.470156 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.469936 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc7rt\" (UniqueName: \"kubernetes.io/projected/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-kube-api-access-vc7rt\") pod \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " Sep 15 22:02:14.470274 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.470239 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-home" (OuterVolumeSpecName: "home") pod "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" (UID: "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:02:14.470423 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.470393 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-model-cache\") pod \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\" (UID: \"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12\") " Sep 15 22:02:14.470646 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.470615 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" (UID: "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:02:14.470989 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.470969 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:02:14.471082 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.470996 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:02:14.471082 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.470729 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" (UID: "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:02:14.472818 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.472785 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" (UID: "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:02:14.473033 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.473005 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-dshm" (OuterVolumeSpecName: "dshm") pod "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" (UID: "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:02:14.473284 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.473257 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-kube-api-access-vc7rt" (OuterVolumeSpecName: "kube-api-access-vc7rt") pod "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" (UID: "c9a7e7b2-36ea-4293-ac86-4ac1bd780b12"). InnerVolumeSpecName "kube-api-access-vc7rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:02:14.571607 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.571572 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:02:14.571607 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.571601 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7rt\" (UniqueName: \"kubernetes.io/projected/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-kube-api-access-vc7rt\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:02:14.571607 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.571613 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:02:14.571841 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.571622 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:02:14.902472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.902382 2575 generic.go:356] "Generic (PLEG): container finished" podID="c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" containerID="ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7" exitCode=0 Sep 15 22:02:14.902472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.902425 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" Sep 15 22:02:14.902472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.902430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" event={"ID":"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12","Type":"ContainerDied","Data":"ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7"} Sep 15 22:02:14.902472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.902459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m" event={"ID":"c9a7e7b2-36ea-4293-ac86-4ac1bd780b12","Type":"ContainerDied","Data":"7b20b0da00286a583359b1239636692eab73b91b193ea5826d0a0f2b474f673b"} Sep 15 22:02:14.902472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.902475 2575 scope.go:117] "RemoveContainer" containerID="ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7" Sep 15 22:02:14.911937 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.911918 2575 scope.go:117] "RemoveContainer" containerID="ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7" Sep 15 22:02:14.912246 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:02:14.912226 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7\": container with ID starting with ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7 not found: ID does not exist" containerID="ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7" Sep 15 22:02:14.912331 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.912259 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7"} err="failed to get container status \"ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7\": rpc error: code = NotFound desc = could not find container \"ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7\": container with ID starting with ceb9d120ecc30a7c3bef946a36e4579dede7def7c59175465b64eaefdbcb01b7 not found: ID does not exist" Sep 15 22:02:14.925325 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.925301 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m"] Sep 15 22:02:14.931302 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:14.931281 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0886df/ipp-route-0886df-kserve-59b976464c-8vb8m"] Sep 15 22:02:15.147140 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:02:15.147091 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" path="/var/lib/kubelet/pods/c9a7e7b2-36ea-4293-ac86-4ac1bd780b12/volumes" Sep 15 22:03:08.524031 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.523993 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r"] Sep 15 22:03:08.524542 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.524223 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" podUID="b41390e4-4285-421e-97d8-12d348c1d41f" containerName="maas-api" containerID="cri-o://acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a" gracePeriod=30 Sep 15 22:03:08.773778 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.773738 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 22:03:08.823147 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.823055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6pj8\" (UniqueName: \"kubernetes.io/projected/b41390e4-4285-421e-97d8-12d348c1d41f-kube-api-access-k6pj8\") pod \"b41390e4-4285-421e-97d8-12d348c1d41f\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " Sep 15 22:03:08.823147 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.823113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41390e4-4285-421e-97d8-12d348c1d41f-openshift-service-ca\") pod \"b41390e4-4285-421e-97d8-12d348c1d41f\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " Sep 15 22:03:08.823147 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.823136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-metrics-certs\") pod \"b41390e4-4285-421e-97d8-12d348c1d41f\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " Sep 15 22:03:08.823416 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.823207 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls\") pod \"b41390e4-4285-421e-97d8-12d348c1d41f\" (UID: \"b41390e4-4285-421e-97d8-12d348c1d41f\") " Sep 15 22:03:08.823584 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.823557 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41390e4-4285-421e-97d8-12d348c1d41f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b41390e4-4285-421e-97d8-12d348c1d41f" (UID: "b41390e4-4285-421e-97d8-12d348c1d41f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 22:03:08.825584 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.825555 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b41390e4-4285-421e-97d8-12d348c1d41f" (UID: "b41390e4-4285-421e-97d8-12d348c1d41f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:03:08.825723 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.825629 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b41390e4-4285-421e-97d8-12d348c1d41f" (UID: "b41390e4-4285-421e-97d8-12d348c1d41f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:03:08.825791 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.825730 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41390e4-4285-421e-97d8-12d348c1d41f-kube-api-access-k6pj8" (OuterVolumeSpecName: "kube-api-access-k6pj8") pod "b41390e4-4285-421e-97d8-12d348c1d41f" (UID: "b41390e4-4285-421e-97d8-12d348c1d41f"). InnerVolumeSpecName "kube-api-access-k6pj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:03:08.924198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.924169 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6pj8\" (UniqueName: \"kubernetes.io/projected/b41390e4-4285-421e-97d8-12d348c1d41f-kube-api-access-k6pj8\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:08.924198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.924196 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41390e4-4285-421e-97d8-12d348c1d41f-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:08.924198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.924206 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:08.924422 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:08.924216 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41390e4-4285-421e-97d8-12d348c1d41f-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:09.167058 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.167029 2575 generic.go:356] "Generic (PLEG): container finished" podID="b41390e4-4285-421e-97d8-12d348c1d41f" containerID="acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a" exitCode=0 Sep 15 22:03:09.167198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.167065 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" Sep 15 22:03:09.167198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.167086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" event={"ID":"b41390e4-4285-421e-97d8-12d348c1d41f","Type":"ContainerDied","Data":"acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a"} Sep 15 22:03:09.167198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.167143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r" event={"ID":"b41390e4-4285-421e-97d8-12d348c1d41f","Type":"ContainerDied","Data":"387a0e640291d8001cc2de6e647a06c61e0c7bd0b4a2387d29d38a3c105d4d8d"} Sep 15 22:03:09.167198 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.167169 2575 scope.go:117] "RemoveContainer" containerID="acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a" Sep 15 22:03:09.176207 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.176184 2575 scope.go:117] "RemoveContainer" containerID="acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a" Sep 15 22:03:09.176551 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:03:09.176527 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a\": container with ID starting with acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a not found: ID does not exist" containerID="acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a" Sep 15 22:03:09.176634 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.176564 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a"} err="failed to get container status \"acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a\": rpc error: code = NotFound desc = could not find container \"acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a\": container with ID starting with acd2a32cfbabd63eab77bea74e30205bd68e8f5f99128ae2314326251af1a11a not found: ID does not exist" Sep 15 22:03:09.185354 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.185329 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r"] Sep 15 22:03:09.192128 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:09.192091 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1297a3-59b9b88d8b-wjr4r"] Sep 15 22:03:11.147268 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:11.147237 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b41390e4-4285-421e-97d8-12d348c1d41f" path="/var/lib/kubelet/pods/b41390e4-4285-421e-97d8-12d348c1d41f/volumes" Sep 15 22:03:53.710276 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:53.710241 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-tzchz"] Sep 15 22:03:53.710853 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:53.710465 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7555b55b4d-tzchz" podUID="fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" containerName="manager" containerID="cri-o://d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548" gracePeriod=30 Sep 15 22:03:53.959971 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:53.959947 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 22:03:54.017063 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.016983 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-webhook-cert\") pod \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " Sep 15 22:03:54.017229 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.017077 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-metrics-certs\") pod \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " Sep 15 22:03:54.017229 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.017151 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4sxs\" (UniqueName: \"kubernetes.io/projected/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-kube-api-access-j4sxs\") pod \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\" (UID: \"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627\") " Sep 15 22:03:54.019386 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.019344 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" (UID: "fcf33e80-fe68-46f7-b9f7-14ed1e4e7627"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:03:54.019513 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.019443 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-kube-api-access-j4sxs" (OuterVolumeSpecName: "kube-api-access-j4sxs") pod "fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" (UID: "fcf33e80-fe68-46f7-b9f7-14ed1e4e7627"). InnerVolumeSpecName "kube-api-access-j4sxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:03:54.019619 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.019596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" (UID: "fcf33e80-fe68-46f7-b9f7-14ed1e4e7627"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:03:54.117960 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.117925 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:54.117960 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.117958 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4sxs\" (UniqueName: \"kubernetes.io/projected/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-kube-api-access-j4sxs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:54.118219 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.117974 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627-webhook-cert\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:03:54.388491 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.388411 2575 generic.go:356] "Generic (PLEG): container finished" podID="fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" containerID="d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548" exitCode=0 Sep 15 22:03:54.388491 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.388485 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-tzchz" Sep 15 22:03:54.388679 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.388484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-tzchz" event={"ID":"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627","Type":"ContainerDied","Data":"d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548"} Sep 15 22:03:54.388679 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.388597 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-tzchz" event={"ID":"fcf33e80-fe68-46f7-b9f7-14ed1e4e7627","Type":"ContainerDied","Data":"60b6d253987b42d181b97df9d83ee2c7d7d3d5c1e2e251c2518e592353b82554"} Sep 15 22:03:54.388679 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.388624 2575 scope.go:117] "RemoveContainer" containerID="d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548" Sep 15 22:03:54.399159 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.399139 2575 scope.go:117] "RemoveContainer" containerID="d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548" Sep 15 22:03:54.399429 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:03:54.399411 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548\": container with ID starting with d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548 not found: ID does not exist" containerID="d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548" Sep 15 22:03:54.399479 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.399436 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548"} err="failed to get container status \"d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548\": rpc error: code = NotFound desc = could not find container \"d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548\": container with ID starting with d113ba797ef958ada6f916ebcc7a3f187b060e7b80125c84b2138d09db40e548 not found: ID does not exist" Sep 15 22:03:54.411333 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.411311 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-tzchz"] Sep 15 22:03:54.416260 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:54.416239 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-tzchz"] Sep 15 22:03:55.147044 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:03:55.147012 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" path="/var/lib/kubelet/pods/fcf33e80-fe68-46f7-b9f7-14ed1e4e7627/volumes" Sep 15 22:04:55.216204 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216175 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7555b55b4d-r58ft"] Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216555 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b41390e4-4285-421e-97d8-12d348c1d41f" containerName="maas-api" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216568 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b41390e4-4285-421e-97d8-12d348c1d41f" containerName="maas-api" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216577 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf82155e-fad3-47e0-a65a-c94b0195d2f8" containerName="maas-api" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216582 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf82155e-fad3-47e0-a65a-c94b0195d2f8" containerName="maas-api" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216592 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" containerName="main" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216598 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" containerName="main" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216613 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" containerName="maas-api" Sep 15 22:04:55.216620 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216620 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" containerName="maas-api" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216626 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" containerName="manager" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216631 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" containerName="manager" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216682 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a7e7b2-36ea-4293-ac86-4ac1bd780b12" containerName="main" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216689 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf82155e-fad3-47e0-a65a-c94b0195d2f8" containerName="maas-api" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216695 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bb5770-6b6b-4581-8a8b-15fa5e1ad395" containerName="maas-api" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216703 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf33e80-fe68-46f7-b9f7-14ed1e4e7627" containerName="manager" Sep 15 22:04:55.216879 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.216709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b41390e4-4285-421e-97d8-12d348c1d41f" containerName="maas-api" Sep 15 22:04:55.220941 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.220923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.224019 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.223996 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 22:04:55.224931 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.224916 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 22:04:55.225303 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.225287 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 22:04:55.225349 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.225320 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svx2b\"" Sep 15 22:04:55.237285 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.237264 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-r58ft"] Sep 15 22:04:55.311827 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.311796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-metrics-certs\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.311980 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.311848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-webhook-cert\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.311980 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.311929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhxj\" (UniqueName: \"kubernetes.io/projected/092a327c-e07b-4a6c-8900-6cecdc181d85-kube-api-access-8zhxj\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.412313 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.412283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhxj\" (UniqueName: \"kubernetes.io/projected/092a327c-e07b-4a6c-8900-6cecdc181d85-kube-api-access-8zhxj\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.412441 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.412341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-metrics-certs\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.412441 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.412377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-webhook-cert\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.414952 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.414924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-metrics-certs\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.415085 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.415064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-webhook-cert\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.421263 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.421238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhxj\" (UniqueName: \"kubernetes.io/projected/092a327c-e07b-4a6c-8900-6cecdc181d85-kube-api-access-8zhxj\") pod \"maas-controller-7555b55b4d-r58ft\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.530512 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.530422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:55.805343 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.805317 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-r58ft"] Sep 15 22:04:55.806709 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:04:55.806686 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092a327c_e07b_4a6c_8900_6cecdc181d85.slice/crio-22738b15ce460008cb5e4da393f1c2188b842d282d65da0c1127938bc1c002bb WatchSource:0}: Error finding container 22738b15ce460008cb5e4da393f1c2188b842d282d65da0c1127938bc1c002bb: Status 404 returned error can't find the container with id 22738b15ce460008cb5e4da393f1c2188b842d282d65da0c1127938bc1c002bb Sep 15 22:04:55.808353 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:55.808336 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 22:04:56.681749 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:56.681714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-r58ft" event={"ID":"092a327c-e07b-4a6c-8900-6cecdc181d85","Type":"ContainerStarted","Data":"22738b15ce460008cb5e4da393f1c2188b842d282d65da0c1127938bc1c002bb"} Sep 15 22:04:57.688768 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:57.688726 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-r58ft" event={"ID":"092a327c-e07b-4a6c-8900-6cecdc181d85","Type":"ContainerStarted","Data":"8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648"} Sep 15 22:04:57.689183 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:57.688811 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:04:57.709042 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:04:57.708984 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7555b55b4d-r58ft" podStartSLOduration=1.696183365 podStartE2EDuration="2.708968141s" totalImagesPullingTime="1.012784776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:04:57.707289818 +0000 UTC m=+2347.046769272" watchObservedRunningTime="2026-09-15 22:04:57.708968141 +0000 UTC m=+2347.048447592" Sep 15 22:05:08.696091 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:08.696052 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:05:15.942361 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:15.942326 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-r58ft"] Sep 15 22:05:15.942779 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:15.942531 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7555b55b4d-r58ft" podUID="092a327c-e07b-4a6c-8900-6cecdc181d85" containerName="manager" containerID="cri-o://8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648" gracePeriod=30 Sep 15 22:05:16.191540 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.191518 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:05:16.293979 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.293894 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-metrics-certs\") pod \"092a327c-e07b-4a6c-8900-6cecdc181d85\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " Sep 15 22:05:16.293979 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.293966 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-webhook-cert\") pod \"092a327c-e07b-4a6c-8900-6cecdc181d85\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " Sep 15 22:05:16.294263 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.294009 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zhxj\" (UniqueName: \"kubernetes.io/projected/092a327c-e07b-4a6c-8900-6cecdc181d85-kube-api-access-8zhxj\") pod \"092a327c-e07b-4a6c-8900-6cecdc181d85\" (UID: \"092a327c-e07b-4a6c-8900-6cecdc181d85\") " Sep 15 22:05:16.296217 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.296183 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "092a327c-e07b-4a6c-8900-6cecdc181d85" (UID: "092a327c-e07b-4a6c-8900-6cecdc181d85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:05:16.296316 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.296221 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092a327c-e07b-4a6c-8900-6cecdc181d85-kube-api-access-8zhxj" (OuterVolumeSpecName: "kube-api-access-8zhxj") pod "092a327c-e07b-4a6c-8900-6cecdc181d85" (UID: "092a327c-e07b-4a6c-8900-6cecdc181d85"). InnerVolumeSpecName "kube-api-access-8zhxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:05:16.296316 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.296216 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "092a327c-e07b-4a6c-8900-6cecdc181d85" (UID: "092a327c-e07b-4a6c-8900-6cecdc181d85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:05:16.395418 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.395390 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:05:16.395418 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.395413 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/092a327c-e07b-4a6c-8900-6cecdc181d85-webhook-cert\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:05:16.395644 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.395427 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhxj\" (UniqueName: \"kubernetes.io/projected/092a327c-e07b-4a6c-8900-6cecdc181d85-kube-api-access-8zhxj\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:05:16.779844 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.779805 2575 generic.go:356] "Generic (PLEG): container finished" podID="092a327c-e07b-4a6c-8900-6cecdc181d85" containerID="8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648" exitCode=0 Sep 15 22:05:16.780019 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.779849 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-r58ft" Sep 15 22:05:16.780019 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.779887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-r58ft" event={"ID":"092a327c-e07b-4a6c-8900-6cecdc181d85","Type":"ContainerDied","Data":"8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648"} Sep 15 22:05:16.780019 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.779942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-r58ft" event={"ID":"092a327c-e07b-4a6c-8900-6cecdc181d85","Type":"ContainerDied","Data":"22738b15ce460008cb5e4da393f1c2188b842d282d65da0c1127938bc1c002bb"} Sep 15 22:05:16.780019 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.779964 2575 scope.go:117] "RemoveContainer" containerID="8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648" Sep 15 22:05:16.792552 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.792531 2575 scope.go:117] "RemoveContainer" containerID="8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648" Sep 15 22:05:16.792840 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:05:16.792821 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648\": container with ID starting with 8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648 not found: ID does not exist" containerID="8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648" Sep 15 22:05:16.792901 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.792850 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648"} err="failed to get container status \"8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648\": rpc error: code = NotFound desc = could not find container \"8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648\": container with ID starting with 8c0ea735f07e703758fa07d0bbeb1e0de223af3c4d7ef646bc0d9549b71b1648 not found: ID does not exist" Sep 15 22:05:16.806675 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.806650 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-r58ft"] Sep 15 22:05:16.812453 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:16.812429 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-r58ft"] Sep 15 22:05:17.147446 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.147361 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092a327c-e07b-4a6c-8900-6cecdc181d85" path="/var/lib/kubelet/pods/092a327c-e07b-4a6c-8900-6cecdc181d85/volumes" Sep 15 22:05:17.756215 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.756180 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7555b55b4d-j7nps"] Sep 15 22:05:17.756607 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.756587 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092a327c-e07b-4a6c-8900-6cecdc181d85" containerName="manager" Sep 15 22:05:17.756689 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.756609 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="092a327c-e07b-4a6c-8900-6cecdc181d85" containerName="manager" Sep 15 22:05:17.756745 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.756718 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="092a327c-e07b-4a6c-8900-6cecdc181d85" containerName="manager" Sep 15 22:05:17.761084 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.761063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:17.763651 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.763624 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 22:05:17.764835 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.764813 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svx2b\"" Sep 15 22:05:17.764966 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.764862 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 22:05:17.764966 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.764913 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 22:05:17.769740 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.769716 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-j7nps"] Sep 15 22:05:17.908366 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.908333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6w75\" (UniqueName: \"kubernetes.io/projected/d002b515-3085-4fcf-938b-f50284385e70-kube-api-access-s6w75\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:17.908578 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.908396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d002b515-3085-4fcf-938b-f50284385e70-metrics-certs\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:17.908578 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:17.908459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d002b515-3085-4fcf-938b-f50284385e70-webhook-cert\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.009306 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.009222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d002b515-3085-4fcf-938b-f50284385e70-metrics-certs\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.009306 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.009259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d002b515-3085-4fcf-938b-f50284385e70-webhook-cert\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.009482 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.009312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6w75\" (UniqueName: \"kubernetes.io/projected/d002b515-3085-4fcf-938b-f50284385e70-kube-api-access-s6w75\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.011858 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.011826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d002b515-3085-4fcf-938b-f50284385e70-webhook-cert\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.011983 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.011876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d002b515-3085-4fcf-938b-f50284385e70-metrics-certs\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.017552 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.017528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6w75\" (UniqueName: \"kubernetes.io/projected/d002b515-3085-4fcf-938b-f50284385e70-kube-api-access-s6w75\") pod \"maas-controller-7555b55b4d-j7nps\" (UID: \"d002b515-3085-4fcf-938b-f50284385e70\") " pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.073569 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.073539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:18.344309 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.344280 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7555b55b4d-j7nps"] Sep 15 22:05:18.346766 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:05:18.346741 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd002b515_3085_4fcf_938b_f50284385e70.slice/crio-00303d8377a436c61bbbb125578cf7695b34c33df0d2d3e4b191ece51c6229cd WatchSource:0}: Error finding container 00303d8377a436c61bbbb125578cf7695b34c33df0d2d3e4b191ece51c6229cd: Status 404 returned error can't find the container with id 00303d8377a436c61bbbb125578cf7695b34c33df0d2d3e4b191ece51c6229cd Sep 15 22:05:18.792843 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:18.792814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-j7nps" event={"ID":"d002b515-3085-4fcf-938b-f50284385e70","Type":"ContainerStarted","Data":"00303d8377a436c61bbbb125578cf7695b34c33df0d2d3e4b191ece51c6229cd"} Sep 15 22:05:19.799165 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:19.799123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7555b55b4d-j7nps" event={"ID":"d002b515-3085-4fcf-938b-f50284385e70","Type":"ContainerStarted","Data":"0220413b49e09921d222c46d2530443109228835ef643593cebf231d2c91bcda"} Sep 15 22:05:19.799582 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:19.799318 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:05:19.816868 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:19.816815 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7555b55b4d-j7nps" podStartSLOduration=2.421232469 podStartE2EDuration="2.81680029s" totalImagesPullingTime="395.567821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:05:19.815509354 +0000 UTC m=+2369.154988801" watchObservedRunningTime="2026-09-15 22:05:19.81680029 +0000 UTC m=+2369.156279740" Sep 15 22:05:30.806864 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:05:30.806834 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7555b55b4d-j7nps" Sep 15 22:06:30.813220 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:30.813128 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z"] Sep 15 22:06:30.813716 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:30.813431 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" containerID="cri-o://842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8" gracePeriod=10 Sep 15 22:06:31.569342 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.569316 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 22:06:31.644015 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.643939 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020d892-df37-466b-8134-3093e91d366b-extensions-socket-volume\") pod \"2020d892-df37-466b-8134-3093e91d366b\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " Sep 15 22:06:31.644015 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.643989 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf8vm\" (UniqueName: \"kubernetes.io/projected/2020d892-df37-466b-8134-3093e91d366b-kube-api-access-rf8vm\") pod \"2020d892-df37-466b-8134-3093e91d366b\" (UID: \"2020d892-df37-466b-8134-3093e91d366b\") " Sep 15 22:06:31.644358 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.644335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2020d892-df37-466b-8134-3093e91d366b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2020d892-df37-466b-8134-3093e91d366b" (UID: "2020d892-df37-466b-8134-3093e91d366b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:06:31.646280 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.646255 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2020d892-df37-466b-8134-3093e91d366b-kube-api-access-rf8vm" (OuterVolumeSpecName: "kube-api-access-rf8vm") pod "2020d892-df37-466b-8134-3093e91d366b" (UID: "2020d892-df37-466b-8134-3093e91d366b"). InnerVolumeSpecName "kube-api-access-rf8vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:06:31.744706 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.744681 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020d892-df37-466b-8134-3093e91d366b-extensions-socket-volume\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:06:31.744706 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:31.744701 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf8vm\" (UniqueName: \"kubernetes.io/projected/2020d892-df37-466b-8134-3093e91d366b-kube-api-access-rf8vm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:06:32.152556 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.152521 2575 generic.go:356] "Generic (PLEG): container finished" podID="2020d892-df37-466b-8134-3093e91d366b" containerID="842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8" exitCode=0 Sep 15 22:06:32.152992 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.152561 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" Sep 15 22:06:32.152992 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.152600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" event={"ID":"2020d892-df37-466b-8134-3093e91d366b","Type":"ContainerDied","Data":"842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8"} Sep 15 22:06:32.152992 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.152648 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z" event={"ID":"2020d892-df37-466b-8134-3093e91d366b","Type":"ContainerDied","Data":"a97cc4f4e7dc90216d928ba460ae0329ee3bcfb6022bc1d1f0cff9496921dcdb"} Sep 15 22:06:32.152992 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.152668 2575 scope.go:117] "RemoveContainer" containerID="842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8" Sep 15 22:06:32.162639 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.162613 2575 scope.go:117] "RemoveContainer" containerID="ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4" Sep 15 22:06:32.171045 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.171029 2575 scope.go:117] "RemoveContainer" containerID="842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8" Sep 15 22:06:32.171339 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:06:32.171320 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8\": container with ID starting with 842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8 not found: ID does not exist" containerID="842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8" Sep 15 22:06:32.171411 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.171348 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8"} err="failed to get container status \"842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8\": rpc error: code = NotFound desc = could not find container \"842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8\": container with ID starting with 842d334873b4fda4ddbfae50b8dd8aff36666c2720a30fb261b637a800a02bf8 not found: ID does not exist" Sep 15 22:06:32.171411 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.171365 2575 scope.go:117] "RemoveContainer" containerID="ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4" Sep 15 22:06:32.171644 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:06:32.171624 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4\": container with ID starting with ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4 not found: ID does not exist" containerID="ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4" Sep 15 22:06:32.171698 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.171651 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4"} err="failed to get container status \"ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4\": rpc error: code = NotFound desc = could not find container \"ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4\": container with ID starting with ec7aca30f0cb8ccc8a75622023a6fdbe932e0af37d92ff709a26da2f316291b4 not found: ID does not exist" Sep 15 22:06:32.174313 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.174289 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z"] Sep 15 22:06:32.177889 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:32.177860 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9qw2z"] Sep 15 22:06:33.147609 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:06:33.147580 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2020d892-df37-466b-8134-3093e91d366b" path="/var/lib/kubelet/pods/2020d892-df37-466b-8134-3093e91d366b/volumes" Sep 15 22:07:31.520958 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.520924 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7"] Sep 15 22:07:31.521374 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.521283 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" Sep 15 22:07:31.521374 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.521294 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" Sep 15 22:07:31.521374 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.521306 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" Sep 15 22:07:31.521374 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.521313 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" Sep 15 22:07:31.521374 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.521376 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" Sep 15 22:07:31.524281 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.524265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.527264 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.527241 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-52mqn\"" Sep 15 22:07:31.542280 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.542257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7"] Sep 15 22:07:31.547680 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.547659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltg7\" (UniqueName: \"kubernetes.io/projected/57a0a7fc-16d6-4498-b090-d7258babc52a-kube-api-access-kltg7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6grr7\" (UID: \"57a0a7fc-16d6-4498-b090-d7258babc52a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.547780 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.547698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57a0a7fc-16d6-4498-b090-d7258babc52a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6grr7\" (UID: \"57a0a7fc-16d6-4498-b090-d7258babc52a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.648227 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.648194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltg7\" (UniqueName: \"kubernetes.io/projected/57a0a7fc-16d6-4498-b090-d7258babc52a-kube-api-access-kltg7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6grr7\" (UID: \"57a0a7fc-16d6-4498-b090-d7258babc52a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.648414 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.648243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57a0a7fc-16d6-4498-b090-d7258babc52a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6grr7\" (UID: \"57a0a7fc-16d6-4498-b090-d7258babc52a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.648613 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.648594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57a0a7fc-16d6-4498-b090-d7258babc52a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6grr7\" (UID: \"57a0a7fc-16d6-4498-b090-d7258babc52a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.657528 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.657504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltg7\" (UniqueName: \"kubernetes.io/projected/57a0a7fc-16d6-4498-b090-d7258babc52a-kube-api-access-kltg7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6grr7\" (UID: \"57a0a7fc-16d6-4498-b090-d7258babc52a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:31.834682 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:31.834593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:32.115797 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:32.115767 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7"] Sep 15 22:07:32.118429 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:07:32.118401 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a0a7fc_16d6_4498_b090_d7258babc52a.slice/crio-d04dbb5698d3594cb5ae560af48db2f2590a659b1449b45853e066ab2a101499 WatchSource:0}: Error finding container d04dbb5698d3594cb5ae560af48db2f2590a659b1449b45853e066ab2a101499: Status 404 returned error can't find the container with id d04dbb5698d3594cb5ae560af48db2f2590a659b1449b45853e066ab2a101499 Sep 15 22:07:32.427896 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:32.427856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" event={"ID":"57a0a7fc-16d6-4498-b090-d7258babc52a","Type":"ContainerStarted","Data":"a516346c31711c0e9ac74d30ba699954131cb057a34c88b131935355847563f8"} Sep 15 22:07:32.428056 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:32.427919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" event={"ID":"57a0a7fc-16d6-4498-b090-d7258babc52a","Type":"ContainerStarted","Data":"d04dbb5698d3594cb5ae560af48db2f2590a659b1449b45853e066ab2a101499"} Sep 15 22:07:32.428056 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:32.427945 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:07:32.454899 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:32.454848 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" podStartSLOduration=1.454833276 podStartE2EDuration="1.454833276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:07:32.452873432 +0000 UTC m=+2501.792352884" watchObservedRunningTime="2026-09-15 22:07:32.454833276 +0000 UTC m=+2501.794312726" Sep 15 22:07:43.434674 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:07:43.434643 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6grr7" Sep 15 22:08:54.166135 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.166076 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt"] Sep 15 22:08:54.166565 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.166551 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2020d892-df37-466b-8134-3093e91d366b" containerName="manager" Sep 15 22:08:54.169442 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.169428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.173594 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.173574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fbb927c8-kserve-self-signed-certs\"" Sep 15 22:08:54.181734 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.181713 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt"] Sep 15 22:08:54.250940 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.250906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-home\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.251125 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.250948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tls-certs\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.251125 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.251026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrbh\" (UniqueName: \"kubernetes.io/projected/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-kube-api-access-txrbh\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.251215 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.251169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-dshm\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.251215 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.251200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-model-cache\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.251281 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.251235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tmp-dir\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.351977 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.351940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-dshm\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.351977 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.351985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-model-cache\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352262 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tmp-dir\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352262 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-home\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352262 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tls-certs\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352409 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrbh\" (UniqueName: \"kubernetes.io/projected/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-kube-api-access-txrbh\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tmp-dir\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352539 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-model-cache\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.352591 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.352569 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-home\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.354472 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.354448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-dshm\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.354952 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.354931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tls-certs\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.365861 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.365830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrbh\" (UniqueName: \"kubernetes.io/projected/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-kube-api-access-txrbh\") pod \"e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.478237 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.478135 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:08:54.753126 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:08:54.753083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b6320f5_dd81_4a69_af06_edc0f3fbd54b.slice/crio-d380df20dfb96ca78a4aa4a81b2ced7e71c3a9707cf2c2b22516e40deeae8bff WatchSource:0}: Error finding container d380df20dfb96ca78a4aa4a81b2ced7e71c3a9707cf2c2b22516e40deeae8bff: Status 404 returned error can't find the container with id d380df20dfb96ca78a4aa4a81b2ced7e71c3a9707cf2c2b22516e40deeae8bff Sep 15 22:08:54.753743 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.753721 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt"] Sep 15 22:08:54.825651 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.825622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" event={"ID":"9b6320f5-dd81-4a69-af06-edc0f3fbd54b","Type":"ContainerStarted","Data":"7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb"} Sep 15 22:08:54.825759 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.825667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" event={"ID":"9b6320f5-dd81-4a69-af06-edc0f3fbd54b","Type":"ContainerStarted","Data":"d380df20dfb96ca78a4aa4a81b2ced7e71c3a9707cf2c2b22516e40deeae8bff"} Sep 15 22:08:54.847153 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:08:54.847090 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" podStartSLOduration=0.847072144 podStartE2EDuration="847.072144ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:08:54.844782346 +0000 UTC m=+2584.184261797" watchObservedRunningTime="2026-09-15 22:08:54.847072144 +0000 UTC m=+2584.186551595" Sep 15 22:09:04.478709 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:04.478668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:09:04.479346 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:04.478825 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:09:04.491778 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:04.491756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:09:04.883986 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:04.883896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:09:11.062481 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:11.062447 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt"] Sep 15 22:09:26.082579 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.082458 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" podUID="9b6320f5-dd81-4a69-af06-edc0f3fbd54b" containerName="main" containerID="cri-o://7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb" gracePeriod=45 Sep 15 22:09:26.336732 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.336673 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:09:26.428332 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428305 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tls-certs\") pod \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " Sep 15 22:09:26.428504 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428350 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txrbh\" (UniqueName: \"kubernetes.io/projected/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-kube-api-access-txrbh\") pod \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " Sep 15 22:09:26.428504 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428377 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-dshm\") pod \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " Sep 15 22:09:26.428504 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428485 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tmp-dir\") pod \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " Sep 15 22:09:26.428685 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428538 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-home\") pod \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " Sep 15 22:09:26.428685 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428580 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-model-cache\") pod \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\" (UID: \"9b6320f5-dd81-4a69-af06-edc0f3fbd54b\") " Sep 15 22:09:26.428799 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428764 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b6320f5-dd81-4a69-af06-edc0f3fbd54b" (UID: "9b6320f5-dd81-4a69-af06-edc0f3fbd54b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:09:26.428894 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-home" (OuterVolumeSpecName: "home") pod "9b6320f5-dd81-4a69-af06-edc0f3fbd54b" (UID: "9b6320f5-dd81-4a69-af06-edc0f3fbd54b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:09:26.428894 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.428877 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b6320f5-dd81-4a69-af06-edc0f3fbd54b" (UID: "9b6320f5-dd81-4a69-af06-edc0f3fbd54b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:09:26.429065 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.429028 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tmp-dir\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:26.429065 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.429047 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-home\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:26.429065 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.429057 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-model-cache\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:26.430843 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.430817 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b6320f5-dd81-4a69-af06-edc0f3fbd54b" (UID: "9b6320f5-dd81-4a69-af06-edc0f3fbd54b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:09:26.430843 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.430828 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-dshm" (OuterVolumeSpecName: "dshm") pod "9b6320f5-dd81-4a69-af06-edc0f3fbd54b" (UID: "9b6320f5-dd81-4a69-af06-edc0f3fbd54b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 22:09:26.430984 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.430868 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-kube-api-access-txrbh" (OuterVolumeSpecName: "kube-api-access-txrbh") pod "9b6320f5-dd81-4a69-af06-edc0f3fbd54b" (UID: "9b6320f5-dd81-4a69-af06-edc0f3fbd54b"). InnerVolumeSpecName "kube-api-access-txrbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:09:26.529952 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.529919 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txrbh\" (UniqueName: \"kubernetes.io/projected/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-kube-api-access-txrbh\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:26.529952 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.529948 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-dshm\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:26.529952 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.529958 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6320f5-dd81-4a69-af06-edc0f3fbd54b-tls-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:26.979131 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.979079 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b6320f5-dd81-4a69-af06-edc0f3fbd54b" containerID="7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb" exitCode=0 Sep 15 22:09:26.979337 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.979162 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" Sep 15 22:09:26.979337 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.979168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" event={"ID":"9b6320f5-dd81-4a69-af06-edc0f3fbd54b","Type":"ContainerDied","Data":"7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb"} Sep 15 22:09:26.979337 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.979214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt" event={"ID":"9b6320f5-dd81-4a69-af06-edc0f3fbd54b","Type":"ContainerDied","Data":"d380df20dfb96ca78a4aa4a81b2ced7e71c3a9707cf2c2b22516e40deeae8bff"} Sep 15 22:09:26.979337 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.979235 2575 scope.go:117] "RemoveContainer" containerID="7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb" Sep 15 22:09:26.988764 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.988749 2575 scope.go:117] "RemoveContainer" containerID="7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb" Sep 15 22:09:26.989018 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:09:26.988997 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb\": container with ID starting with 7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb not found: ID does not exist" containerID="7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb" Sep 15 22:09:26.989064 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:26.989027 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb"} err="failed to get container status \"7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb\": rpc error: code = NotFound desc = could not find container \"7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb\": container with ID starting with 7ee3e5019c5ebf7c5206182a4f6ce63895d60af4a3f355aa8a56aa5f44d67bdb not found: ID does not exist" Sep 15 22:09:27.005152 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:27.005126 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt"] Sep 15 22:09:27.011172 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:27.011151 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fbb927c8-kserve-7f67649895-rzpxt"] Sep 15 22:09:27.146761 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:27.146727 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6320f5-dd81-4a69-af06-edc0f3fbd54b" path="/var/lib/kubelet/pods/9b6320f5-dd81-4a69-af06-edc0f3fbd54b/volumes" Sep 15 22:09:34.775370 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.775318 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x"] Sep 15 22:09:34.775864 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.775689 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6320f5-dd81-4a69-af06-edc0f3fbd54b" containerName="main" Sep 15 22:09:34.775864 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.775702 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6320f5-dd81-4a69-af06-edc0f3fbd54b" containerName="main" Sep 15 22:09:34.775864 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.775787 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6320f5-dd81-4a69-af06-edc0f3fbd54b" containerName="main" Sep 15 22:09:34.778444 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.778424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:34.804338 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.804311 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x"] Sep 15 22:09:34.907349 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.907307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-metrics-certs\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:34.907548 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.907375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw84\" (UniqueName: \"kubernetes.io/projected/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-kube-api-access-fdw84\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:34.907548 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.907423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-maas-api-tls\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:34.907548 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:34.907537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-openshift-service-ca\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.008705 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.008671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw84\" (UniqueName: \"kubernetes.io/projected/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-kube-api-access-fdw84\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.008893 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.008721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-maas-api-tls\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.008893 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.008766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-openshift-service-ca\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.008893 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.008787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-metrics-certs\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.009487 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.009460 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-openshift-service-ca\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.011484 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.011454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-metrics-certs\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.011562 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.011454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-maas-api-tls\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.018003 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.017977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw84\" (UniqueName: \"kubernetes.io/projected/15ecc6f0-16fc-4ba1-a672-30d8edfc9d19-kube-api-access-fdw84\") pod \"maas-api-5bbfc7d884-qrk6x\" (UID: \"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19\") " pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.089218 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.089140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:35.363685 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:35.363662 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x"] Sep 15 22:09:35.365714 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:09:35.365685 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ecc6f0_16fc_4ba1_a672_30d8edfc9d19.slice/crio-de1379575a99a2eda3da4bf04be59f7b41680fcf9e388c7d11a2911267cca320 WatchSource:0}: Error finding container de1379575a99a2eda3da4bf04be59f7b41680fcf9e388c7d11a2911267cca320: Status 404 returned error can't find the container with id de1379575a99a2eda3da4bf04be59f7b41680fcf9e388c7d11a2911267cca320 Sep 15 22:09:36.024419 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:36.024383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" event={"ID":"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19","Type":"ContainerStarted","Data":"de1379575a99a2eda3da4bf04be59f7b41680fcf9e388c7d11a2911267cca320"} Sep 15 22:09:37.030969 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:37.030931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" event={"ID":"15ecc6f0-16fc-4ba1-a672-30d8edfc9d19","Type":"ContainerStarted","Data":"d75436f4306ab5484c465c558a3bb720b01ef1c59ca7fd5c60ae08f82c9c5b0d"} Sep 15 22:09:37.031393 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:37.030999 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:37.061277 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:37.061229 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" podStartSLOduration=2.446821157 podStartE2EDuration="3.061212824s" totalImagesPullingTime="614.391667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:09:37.058715642 +0000 UTC m=+2626.398195092" watchObservedRunningTime="2026-09-15 22:09:37.061212824 +0000 UTC m=+2626.400692275" Sep 15 22:09:43.040980 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:43.040952 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bbfc7d884-qrk6x" Sep 15 22:09:43.112326 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:43.112295 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs"] Sep 15 22:09:43.112529 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:43.112504 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" podUID="c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" containerName="maas-api" containerID="cri-o://cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56" gracePeriod=30 Sep 15 22:09:44.359456 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.359431 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 22:09:44.494988 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.494903 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-metrics-certs\") pod \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " Sep 15 22:09:44.494988 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.494961 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls\") pod \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " Sep 15 22:09:44.495228 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.495009 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-openshift-service-ca\") pod \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " Sep 15 22:09:44.495228 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.495027 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llpxv\" (UniqueName: \"kubernetes.io/projected/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-kube-api-access-llpxv\") pod \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\" (UID: \"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05\") " Sep 15 22:09:44.495367 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.495342 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" (UID: "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 22:09:44.497250 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.497220 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-kube-api-access-llpxv" (OuterVolumeSpecName: "kube-api-access-llpxv") pod "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" (UID: "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05"). InnerVolumeSpecName "kube-api-access-llpxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 22:09:44.497368 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.497262 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" (UID: "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:09:44.497368 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.497288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" (UID: "c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 22:09:44.595999 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.595968 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-metrics-certs\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:44.595999 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.595993 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-maas-api-tls\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:44.595999 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.596003 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-openshift-service-ca\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:44.596264 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:44.596014 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llpxv\" (UniqueName: \"kubernetes.io/projected/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05-kube-api-access-llpxv\") on node \"ip-10-0-130-52.ec2.internal\" DevicePath \"\"" Sep 15 22:09:45.068863 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.068826 2575 generic.go:356] "Generic (PLEG): container finished" podID="c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" containerID="cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56" exitCode=0 Sep 15 22:09:45.069040 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.068877 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" Sep 15 22:09:45.069040 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.068911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" event={"ID":"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05","Type":"ContainerDied","Data":"cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56"} Sep 15 22:09:45.069040 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.068967 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs" event={"ID":"c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05","Type":"ContainerDied","Data":"a4a6593dc6da1088324280453699210f052eed9669ca9154ac6fc53635ef5b13"} Sep 15 22:09:45.069040 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.068991 2575 scope.go:117] "RemoveContainer" containerID="cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56" Sep 15 22:09:45.078531 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.078513 2575 scope.go:117] "RemoveContainer" containerID="cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56" Sep 15 22:09:45.078842 ip-10-0-130-52 kubenswrapper[2575]: E0915 22:09:45.078824 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56\": container with ID starting with cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56 not found: ID does not exist" containerID="cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56" Sep 15 22:09:45.078896 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.078850 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56"} err="failed to get container status \"cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56\": rpc error: code = NotFound desc = could not find container \"cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56\": container with ID starting with cb407457047f90ac8fd4933e2352f179ff54e171ff8d06fc898c059f91eccc56 not found: ID does not exist" Sep 15 22:09:45.092944 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.092917 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs"] Sep 15 22:09:45.101831 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.101810 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5557cbddd4-cvfxs"] Sep 15 22:09:45.152752 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:09:45.152725 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" path="/var/lib/kubelet/pods/c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05/volumes" Sep 15 22:10:05.032575 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:05.032543 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558d748df4-5rq7c_aafbc96c-896a-4a85-aed1-2b56f551a38d/authorino/0.log" Sep 15 22:10:10.966301 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:10.966269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fdd6s_c3642b6f-2bbb-4075-b40c-1ebba6842f57/manager/0.log" Sep 15 22:10:11.100837 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:11.100809 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rdqbv_c582fa99-30ba-455f-b3e4-92ff4fb770e0/manager/0.log" Sep 15 22:10:11.241579 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:11.241497 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7555b55b4d-j7nps_d002b515-3085-4fcf-938b-f50284385e70/manager/0.log" Sep 15 22:10:11.363415 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:11.363385 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jmh62_3b703202-1f21-461a-8791-859ee3a573ba/server/0.log" Sep 15 22:10:11.505250 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:11.505178 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rnkfz_ec606442-9df8-4aa9-9cf5-6953444b7453/manager/0.log" Sep 15 22:10:11.632959 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:11.632932 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-884vc_5e7ab201-4302-420a-b401-6aa2d55b9447/copy-manifests/0.log" Sep 15 22:10:11.642305 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:11.642279 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-884vc_5e7ab201-4302-420a-b401-6aa2d55b9447/manager/0.log" Sep 15 22:10:12.438030 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:12.438000 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bbfc7d884-qrk6x_15ecc6f0-16fc-4ba1-a672-30d8edfc9d19/maas-api/0.log" Sep 15 22:10:12.805322 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:12.805243 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-mc522_b4a2541a-85b3-4899-bf60-4f0c70474ba5/postgres/0.log" Sep 15 22:10:13.659670 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:13.659634 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558d748df4-5rq7c_aafbc96c-896a-4a85-aed1-2b56f551a38d/authorino/0.log" Sep 15 22:10:13.785238 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:13.785212 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5lr4s_fb9954ae-c239-46bc-b12d-9f883a1a6af3/manager/0.log" Sep 15 22:10:13.913239 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:13.913204 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-29tdb_d6ae0828-b03e-4601-86dc-e2f77ec60824/manager/0.log" Sep 15 22:10:14.174449 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:14.174332 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6grr7_57a0a7fc-16d6-4498-b090-d7258babc52a/manager/0.log" Sep 15 22:10:14.458361 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:14.458279 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kch8c_32498c46-5964-4d1d-8f30-5aac8b7245f6/manager/0.log" Sep 15 22:10:15.216434 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:15.216406 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-l48jh_a52160bd-aebe-4ae4-bca5-8bc03a23b7de/kube-auth-proxy/0.log" Sep 15 22:10:16.221116 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:16.221069 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vwn47_02954641-80ba-4628-99a9-bade4aa2f9a8/main/0.log" Sep 15 22:10:16.607160 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:16.607064 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkz4k_d1b88e0a-42b4-489a-8918-b6b7b32a32dd/main/0.log" Sep 15 22:10:16.736212 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:16.736177 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-w4ng5_28ac6010-8a36-4a4b-96a5-4bc8d8ee5302/main/0.log" Sep 15 22:10:25.315142 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:25.315082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d7wjb_b1bb12a7-b2b9-4153-b889-21b5cfe1dc1f/global-pull-secret-syncer/0.log" Sep 15 22:10:25.504617 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:25.504571 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qwbjf_d12fc48c-d6c5-4f9a-9e5b-1c210523b8b9/konnectivity-agent/0.log" Sep 15 22:10:25.566274 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:25.566202 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-52.ec2.internal_0e5612082f84072df6a197a760b84512/haproxy/0.log" Sep 15 22:10:30.021129 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:30.021082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558d748df4-5rq7c_aafbc96c-896a-4a85-aed1-2b56f551a38d/authorino/0.log" Sep 15 22:10:30.078204 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:30.078148 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5lr4s_fb9954ae-c239-46bc-b12d-9f883a1a6af3/manager/0.log" Sep 15 22:10:30.110650 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:30.110616 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-29tdb_d6ae0828-b03e-4601-86dc-e2f77ec60824/manager/0.log" Sep 15 22:10:30.227295 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:30.227263 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6grr7_57a0a7fc-16d6-4498-b090-d7258babc52a/manager/0.log" Sep 15 22:10:30.340311 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:30.340223 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kch8c_32498c46-5964-4d1d-8f30-5aac8b7245f6/manager/0.log" Sep 15 22:10:32.462348 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:32.462325 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdkxr_32fd200d-709f-40db-8b1f-a0db9c35a906/node-exporter/0.log" Sep 15 22:10:32.488226 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:32.488195 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdkxr_32fd200d-709f-40db-8b1f-a0db9c35a906/kube-rbac-proxy/0.log" Sep 15 22:10:32.520642 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:32.520620 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdkxr_32fd200d-709f-40db-8b1f-a0db9c35a906/init-textfile/0.log" Sep 15 22:10:35.587129 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.587079 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777"] Sep 15 22:10:35.587503 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.587446 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" containerName="maas-api" Sep 15 22:10:35.587503 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.587458 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" containerName="maas-api" Sep 15 22:10:35.587594 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.587513 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a42ef4-a4c6-4a38-8667-d1c5b4ce9e05" containerName="maas-api" Sep 15 22:10:35.590587 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.590568 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.593074 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.593048 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgf4x\"/\"openshift-service-ca.crt\"" Sep 15 22:10:35.593209 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.593093 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgf4x\"/\"kube-root-ca.crt\"" Sep 15 22:10:35.593913 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.593898 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lgf4x\"/\"default-dockercfg-5glwm\"" Sep 15 22:10:35.598975 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.598953 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777"] Sep 15 22:10:35.727628 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.727598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-podres\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.727778 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.727645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-sys\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.727778 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.727743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-proc\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.727857 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.727795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m66z\" (UniqueName: \"kubernetes.io/projected/556a2c3e-67af-40da-b7fd-c13bd1351280-kube-api-access-5m66z\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.727857 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.727835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-lib-modules\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828252 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-sys\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828394 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-proc\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828394 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m66z\" (UniqueName: \"kubernetes.io/projected/556a2c3e-67af-40da-b7fd-c13bd1351280-kube-api-access-5m66z\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828394 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-sys\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828394 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-lib-modules\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828529 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-podres\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828529 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-proc\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828529 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828478 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-lib-modules\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.828529 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.828503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/556a2c3e-67af-40da-b7fd-c13bd1351280-podres\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.837318 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.837254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m66z\" (UniqueName: \"kubernetes.io/projected/556a2c3e-67af-40da-b7fd-c13bd1351280-kube-api-access-5m66z\") pod \"perf-node-gather-daemonset-qj777\" (UID: \"556a2c3e-67af-40da-b7fd-c13bd1351280\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:35.900846 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:35.900824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:36.174008 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:36.173983 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777"] Sep 15 22:10:36.176402 ip-10-0-130-52 kubenswrapper[2575]: W0915 22:10:36.176375 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod556a2c3e_67af_40da_b7fd_c13bd1351280.slice/crio-405fe8d7041b9605a738fd3ff23a93eb1902b4531bb2e503a27a6559f40a51cd WatchSource:0}: Error finding container 405fe8d7041b9605a738fd3ff23a93eb1902b4531bb2e503a27a6559f40a51cd: Status 404 returned error can't find the container with id 405fe8d7041b9605a738fd3ff23a93eb1902b4531bb2e503a27a6559f40a51cd Sep 15 22:10:36.301782 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:36.301748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" event={"ID":"556a2c3e-67af-40da-b7fd-c13bd1351280","Type":"ContainerStarted","Data":"4baca001c43d0e65b0f6b905071b20a0de26e5b201a1b6d466460b779db1c759"} Sep 15 22:10:36.301914 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:36.301805 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" event={"ID":"556a2c3e-67af-40da-b7fd-c13bd1351280","Type":"ContainerStarted","Data":"405fe8d7041b9605a738fd3ff23a93eb1902b4531bb2e503a27a6559f40a51cd"} Sep 15 22:10:36.301914 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:36.301833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:36.319406 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:36.319358 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" podStartSLOduration=1.319341326 podStartE2EDuration="1.319341326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:10:36.318553272 +0000 UTC m=+2685.658032720" watchObservedRunningTime="2026-09-15 22:10:36.319341326 +0000 UTC m=+2685.658820777" Sep 15 22:10:37.765439 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:37.765403 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sg6pl_a72d9f28-2893-48a1-9089-8dcaf636f332/dns/0.log" Sep 15 22:10:37.789764 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:37.789736 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sg6pl_a72d9f28-2893-48a1-9089-8dcaf636f332/kube-rbac-proxy/0.log" Sep 15 22:10:37.912014 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:37.911994 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6js4_026a5080-dd49-4fed-8060-427077d7a430/dns-node-resolver/0.log" Sep 15 22:10:38.514719 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:38.514685 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-757c5b4b9d-7pz5c_fdb54a01-af83-4d22-9437-a97cb7169382/registry/0.log" Sep 15 22:10:38.570529 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:38.570494 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tmpbr_ed1d53b4-089f-464e-94dc-20f3887ee86d/node-ca/0.log" Sep 15 22:10:40.462218 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:40.462189 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-l48jh_a52160bd-aebe-4ae4-bca5-8bc03a23b7de/kube-auth-proxy/0.log" Sep 15 22:10:41.536706 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:41.536675 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6rdt7_30c0de63-a264-4c60-8166-f0bdbfca7e0c/serve-healthcheck-canary/0.log" Sep 15 22:10:42.259505 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:42.259476 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2npk_1014602b-273c-4d80-ab31-7794c4186635/kube-rbac-proxy/0.log" Sep 15 22:10:42.293566 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:42.293539 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2npk_1014602b-273c-4d80-ab31-7794c4186635/exporter/0.log" Sep 15 22:10:42.317135 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:42.317112 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-qj777" Sep 15 22:10:42.320629 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:42.320597 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2npk_1014602b-273c-4d80-ab31-7794c4186635/extractor/0.log" Sep 15 22:10:44.797593 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:44.797562 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fdd6s_c3642b6f-2bbb-4075-b40c-1ebba6842f57/manager/0.log" Sep 15 22:10:45.289907 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:45.289867 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rdqbv_c582fa99-30ba-455f-b3e4-92ff4fb770e0/manager/0.log" Sep 15 22:10:45.369496 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:45.369466 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7555b55b4d-j7nps_d002b515-3085-4fcf-938b-f50284385e70/manager/0.log" Sep 15 22:10:45.397073 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:45.396978 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jmh62_3b703202-1f21-461a-8791-859ee3a573ba/server/0.log" Sep 15 22:10:45.536790 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:45.536756 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rnkfz_ec606442-9df8-4aa9-9cf5-6953444b7453/manager/0.log" Sep 15 22:10:45.562269 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:45.562241 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-884vc_5e7ab201-4302-420a-b401-6aa2d55b9447/manager/0.log" Sep 15 22:10:45.587425 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:45.587402 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-884vc_5e7ab201-4302-420a-b401-6aa2d55b9447/copy-manifests/0.log" Sep 15 22:10:46.542156 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:46.542120 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-l48jh_a52160bd-aebe-4ae4-bca5-8bc03a23b7de/kube-auth-proxy/0.log" Sep 15 22:10:47.124812 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:47.124783 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5rbfw_ad56ee4b-84d5-4944-abb9-bcce76bd7c5f/openshift-lws-operator/0.log" Sep 15 22:10:49.148923 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:49.148894 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-l48jh_a52160bd-aebe-4ae4-bca5-8bc03a23b7de/kube-auth-proxy/0.log" Sep 15 22:10:52.220012 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:52.219976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mxkcq_b5cb82f5-20ca-488d-a342-a13a72016458/migrator/0.log" Sep 15 22:10:52.239519 ip-10-0-130-52 kubenswrapper[2575]: I0915 22:10:52.239494 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mxkcq_b5cb82f5-20ca-488d-a342-a13a72016458/graceful-termination/0.log"