Sep 17 17:29:17.026877 ip-10-0-142-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 17:29:17.026890 ip-10-0-142-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 17:29:17.026899 ip-10-0-142-117 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 17:29:17.027092 ip-10-0-142-117 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 17:29:27.060890 ip-10-0-142-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 17:29:27.060912 ip-10-0-142-117 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot aa921002ccfe4a519089601cf65639f3 -- Sep 17 17:31:01.997654 ip-10-0-142-117 systemd[1]: Starting Kubernetes Kubelet... Sep 17 17:31:02.418043 ip-10-0-142-117 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 17:31:02.418043 ip-10-0-142-117 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 17:31:02.418043 ip-10-0-142-117 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 17:31:02.418043 ip-10-0-142-117 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 17:31:02.418043 ip-10-0-142-117 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 17:31:02.419218 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.419131 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 17:31:02.424861 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424846 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:31:02.424861 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424860 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424864 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424868 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424872 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424875 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424878 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424881 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424883 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424886 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424889 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424891 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424894 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424897 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424900 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424902 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424905 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424908 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424910 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424913 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424916 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:31:02.424930 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424919 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424921 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424924 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424928 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424932 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424936 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424939 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424943 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424946 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424949 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424952 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424955 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424958 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424961 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424963 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424967 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424970 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424973 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424975 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424978 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:31:02.425422 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424980 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424983 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424985 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424988 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424990 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424993 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424995 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.424997 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425000 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425002 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425005 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425007 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425010 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425012 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425016 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425018 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425022 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425024 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425029 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:31:02.425905 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425032 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425035 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425038 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425040 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425043 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425045 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425048 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425051 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425053 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425056 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425059 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425062 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425064 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425068 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425071 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425074 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425078 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425081 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425084 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:31:02.426383 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425086 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425089 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425091 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425094 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425097 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425100 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425104 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425106 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425109 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425112 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425114 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425117 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425120 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425122 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425125 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425127 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425130 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425132 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425135 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:31:02.426843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.425138 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:31:02.427544 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427531 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:31:02.427544 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427543 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427546 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427550 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427553 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427556 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427559 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427562 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427564 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427567 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427569 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427572 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427575 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427577 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427580 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427582 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427585 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427588 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427590 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427593 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427595 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:31:02.427604 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427598 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427601 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427604 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427606 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427609 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427612 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427615 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427618 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427621 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427624 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427626 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427629 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427631 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427634 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427636 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427639 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427642 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427645 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427647 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427650 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:31:02.428106 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427652 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427654 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427657 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427660 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427662 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427664 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427667 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427669 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427672 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427675 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427677 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427680 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427682 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427685 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427690 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427694 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427697 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427700 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427703 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:31:02.428894 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427705 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427709 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427712 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427715 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427719 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427722 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427725 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427728 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427730 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427737 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427740 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427743 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427746 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427749 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427752 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427754 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427757 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427760 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427762 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:31:02.429649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427765 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427768 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427770 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427772 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427775 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427777 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427780 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427784 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427788 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427791 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427793 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427796 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427799 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427802 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427804 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427807 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427809 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427812 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427814 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:31:02.430140 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.427817 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428426 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428444 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428453 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428461 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428468 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428482 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428489 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428497 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428505 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428512 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428519 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428526 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428532 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428540 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428547 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428554 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428560 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428568 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428581 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:31:02.430623 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428591 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428598 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428605 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428613 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428620 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428627 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428634 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428642 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428649 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428656 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428663 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428675 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428683 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428690 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428698 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.428705 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429168 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429176 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429181 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:31:02.431188 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429186 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429191 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429195 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429201 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429211 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429215 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429220 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429224 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429228 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429233 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429237 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429242 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429246 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429251 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429255 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429259 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429268 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429273 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429277 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429282 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:31:02.431714 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429290 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429297 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429301 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429305 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429329 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429332 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429348 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429352 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429465 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429469 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429472 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429475 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429478 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429481 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429483 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429486 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429488 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429491 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429494 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:31:02.432202 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429497 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429500 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429503 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429505 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429508 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429511 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429514 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429516 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429519 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429522 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429525 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429527 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429530 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429533 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429537 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429541 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429544 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429548 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429551 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:31:02.432669 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429554 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429556 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.429559 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429640 2566 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429673 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429680 2566 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429685 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429689 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429692 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429697 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429701 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429705 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429708 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429711 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429715 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429719 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429722 2566 flags.go:64] FLAG: --cgroup-root="" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429725 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429728 2566 flags.go:64] FLAG: --client-ca-file="" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429731 2566 flags.go:64] FLAG: --cloud-provider="external" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429734 2566 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429737 2566 flags.go:64] FLAG: --cluster-domain="" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429740 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429744 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 17:31:02.433165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429747 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429750 2566 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429754 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429757 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429760 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429763 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429766 2566 flags.go:64] FLAG: --contention-profiling="false" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429769 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429772 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429775 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429778 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429782 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429785 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429788 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429791 2566 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429794 2566 flags.go:64] FLAG: --enable-server="true" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429797 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429802 2566 flags.go:64] FLAG: --event-burst="100" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429805 2566 flags.go:64] FLAG: --event-qps="50" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429808 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429811 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429814 2566 flags.go:64] FLAG: --eviction-hard="" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429818 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429822 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429825 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 17:31:02.433758 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429828 2566 flags.go:64] FLAG: --eviction-soft="" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429830 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429833 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429836 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429839 2566 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429842 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429846 2566 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429849 2566 flags.go:64] FLAG: --feature-gates="" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429853 2566 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429855 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429859 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429862 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429865 2566 flags.go:64] FLAG: --healthz-port="10248" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429867 2566 flags.go:64] FLAG: --help="false" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429870 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429873 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429876 2566 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429879 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429882 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429886 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429889 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429892 2566 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429894 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 17:31:02.434336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429900 2566 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429903 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429906 2566 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429909 2566 flags.go:64] FLAG: --kube-reserved="" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429914 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429916 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429920 2566 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429923 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429926 2566 flags.go:64] FLAG: --lock-file="" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429928 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429931 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429934 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429940 2566 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429943 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429946 2566 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429949 2566 flags.go:64] FLAG: --logging-format="text" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429952 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429955 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429958 2566 flags.go:64] FLAG: --manifest-url="" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429961 2566 flags.go:64] FLAG: --manifest-url-header="" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429965 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429968 2566 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429972 2566 flags.go:64] FLAG: --max-pods="110" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429975 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429978 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 17:31:02.434954 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429981 2566 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429984 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429987 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429990 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.429992 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430001 2566 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430004 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430008 2566 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430011 2566 flags.go:64] FLAG: --pod-cidr="" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430015 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430021 2566 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430024 2566 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430029 2566 flags.go:64] FLAG: --pods-per-core="0" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430032 2566 flags.go:64] FLAG: --port="10250" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430036 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430039 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07b18f0dd1ca7e64f" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430042 2566 flags.go:64] FLAG: --qos-reserved="" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430045 2566 flags.go:64] FLAG: --read-only-port="10255" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430048 2566 flags.go:64] FLAG: --register-node="true" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430051 2566 flags.go:64] FLAG: --register-with-taints="" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430054 2566 flags.go:64] FLAG: --registry-burst="10" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430057 2566 flags.go:64] FLAG: --registry-qps="5" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430060 2566 flags.go:64] FLAG: --reserved-cpus="" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430063 2566 flags.go:64] FLAG: --reserved-memory="" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430067 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 17:31:02.435578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430071 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430074 2566 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430077 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430079 2566 flags.go:64] FLAG: --runonce="false" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430082 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430085 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430088 2566 flags.go:64] FLAG: --seccomp-default="false" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430091 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430094 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430097 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430100 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430103 2566 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430106 2566 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430109 2566 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430113 2566 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430116 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430119 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430122 2566 flags.go:64] FLAG: --system-cgroups="" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430125 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430132 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430135 2566 flags.go:64] FLAG: --tls-cert-file="" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430138 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430141 2566 flags.go:64] FLAG: --tls-min-version="" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430144 2566 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430147 2566 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 17:31:02.436231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430150 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430153 2566 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430156 2566 flags.go:64] FLAG: --v="2" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430161 2566 flags.go:64] FLAG: --version="false" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430165 2566 flags.go:64] FLAG: --vmodule="" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430169 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.430173 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430281 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430286 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430289 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430292 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430295 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430297 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430300 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430303 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430305 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430308 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430310 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430313 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430316 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430318 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:31:02.436878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430322 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430325 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430328 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430331 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430334 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430337 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430340 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430343 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430345 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430348 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430351 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430353 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430373 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430377 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430381 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430386 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430389 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430393 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430396 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:31:02.437417 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430399 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430402 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430406 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430410 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430412 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430415 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430418 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430420 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430423 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430426 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430428 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430431 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430434 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430438 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430441 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430444 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430446 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430449 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430453 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430456 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:31:02.437875 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430458 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430461 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430464 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430466 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430471 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430474 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430477 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430480 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430483 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430486 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430489 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430492 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430495 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430498 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430500 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430503 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430506 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430509 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430512 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:31:02.438396 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430515 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430519 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430522 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430524 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430527 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430530 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430534 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430536 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430539 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430541 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430544 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430548 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430551 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430554 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430556 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430559 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430562 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430565 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430567 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430570 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:31:02.438872 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430573 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430576 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430578 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430581 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430583 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430587 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.430589 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:31:02.439354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.431328 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 17:31:02.439646 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.439628 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 17:31:02.439678 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.439646 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 17:31:02.441422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.441412 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 17:31:02.441455 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.441423 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 17:31:02.441496 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441481 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:31:02.441496 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441486 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:31:02.441496 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441490 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:31:02.441496 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441493 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:31:02.441496 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441496 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441499 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441502 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441505 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441508 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441511 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441514 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441517 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441520 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441522 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441525 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441528 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441530 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441533 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441536 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441538 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441541 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441544 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441547 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441549 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:31:02.441622 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441552 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441556 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441559 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441561 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441564 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441567 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441569 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441572 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441574 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441577 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441581 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441585 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441588 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441591 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441595 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441599 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441601 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441604 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441607 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:31:02.442137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441611 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441614 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441617 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441620 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441623 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441626 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441629 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441632 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441634 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441637 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441640 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441643 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441645 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441648 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441650 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441654 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441657 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441659 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441662 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441665 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:31:02.442618 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441668 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441670 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441673 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441675 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441678 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441681 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441683 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441688 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441690 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441693 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441697 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441701 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441704 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441707 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441709 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441712 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441715 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441718 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441720 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441723 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:31:02.443100 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441726 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441728 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441731 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441734 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441737 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441739 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441742 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441745 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441748 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441751 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441754 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441756 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441759 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441762 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441765 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:31:02.443593 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441767 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.441772 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441906 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441912 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441916 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441919 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441923 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441927 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441930 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441933 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441937 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441940 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441945 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441949 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:31:02.443972 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441952 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441955 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441957 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441960 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441963 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441966 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441968 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441971 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441973 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441976 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441980 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441983 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441987 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441990 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441994 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.441997 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442000 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442003 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442005 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:31:02.444332 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442008 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442011 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442014 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442017 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442019 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442022 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442025 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442027 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442030 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442033 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442036 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442038 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442041 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442044 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442046 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442049 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442052 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442055 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442057 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442060 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:31:02.444805 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442064 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442066 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442069 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442071 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442074 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442077 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442080 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442083 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442085 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442088 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442090 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442093 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442096 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442099 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442101 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442104 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442107 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442110 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442112 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442115 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:31:02.445286 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442117 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442120 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442123 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442125 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442128 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442131 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442133 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442136 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442139 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442142 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442145 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442147 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442150 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442153 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442155 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442158 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442161 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442163 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442166 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442169 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:31:02.445776 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442172 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442175 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442177 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442181 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442183 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442186 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442189 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:02.442192 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.442197 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.442301 2566 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.444721 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.445784 2566 server.go:1019] "Starting client certificate rotation" Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.445881 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 17:31:02.446244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.445915 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 17:31:02.448880 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.448866 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 17:31:02.450669 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.450651 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 17:31:02.470811 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.470790 2566 log.go:25] "Validated CRI v1 runtime API" Sep 17 17:31:02.475989 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.475971 2566 log.go:25] "Validated CRI v1 image API" Sep 17 17:31:02.476782 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.476765 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 17:31:02.481382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.481345 2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c8a252e-c91e-4a9f-b377-19740714b48c:/dev/nvme0n1p3 9cb69645-42de-4230-b09b-58403b46cfd5:/dev/nvme0n1p4] Sep 17 17:31:02.481471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.481382 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 17:31:02.484858 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.484836 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 17:31:02.486475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.486341 2566 manager.go:216] Machine: {Timestamp:2026-09-17 17:31:02.485039188 +0000 UTC m=+0.374800950 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100024 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c044870f68d1906e860a63332187b SystemUUID:ec2c0448-70f6-8d19-06e8-60a63332187b BootID:aa921002-ccfe-4a51-9089-601cf65639f3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:57:a2:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:57:a2:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:00:de:c7:df:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 17:31:02.486475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.486464 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 17:31:02.486635 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.486575 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 17:31:02.489926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.489896 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 17:31:02.490054 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.489930 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 17:31:02.490714 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.490704 2566 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 17:31:02.490752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.490717 2566 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 17:31:02.490752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.490724 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 17:31:02.491420 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.491409 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 17:31:02.492908 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.492898 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 17:31:02.496912 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.496901 2566 state_mem.go:36] "Initialized new in-memory state store" Sep 17 17:31:02.497019 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497010 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 17:31:02.497056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497039 2566 kubelet.go:486] "Attempting to sync node with API server" Sep 17 17:31:02.497056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497048 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 17:31:02.497112 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497059 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 17:31:02.497112 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497069 2566 kubelet.go:398] "Adding apiserver pod source" Sep 17 17:31:02.497112 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497077 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 17:31:02.497446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497435 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 17:31:02.497499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.497454 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 17:31:02.499737 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.499716 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 17:31:02.502698 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502683 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 17:31:02.502798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502722 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 17:31:02.502949 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502935 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 17:31:02.503007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502956 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 17:31:02.503007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502964 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 17:31:02.503007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502972 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 17:31:02.503007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502983 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 17:31:02.503007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.502992 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 17:31:02.503520 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.503510 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 17:31:02.503571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.503522 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 17:31:02.503571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.503534 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 17:31:02.503571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.503543 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 17:31:02.503571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.503556 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 17:31:02.503571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.503568 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 17:31:02.504446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.504431 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 17:31:02.504446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.504447 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 17:31:02.508340 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.508327 2566 server.go:1268] "Started kubelet" Sep 17 17:31:02.508464 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.508432 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 17:31:02.508558 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.508484 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 17:31:02.508558 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.508552 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 17:31:02.508844 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.508825 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 17:31:02.509139 ip-10-0-142-117 systemd[1]: Started Kubernetes Kubelet. Sep 17 17:31:02.511496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.510628 2566 server.go:310] "Adding debug handlers to kubelet server" Sep 17 17:31:02.513818 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.512937 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 17:31:02.513818 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.512975 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 17:31:02.513818 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.513069 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 17:31:02.515783 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.514158 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 17:31:02.518604 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.517510 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd34eec4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.50830139 +0000 UTC m=+0.398063149,LastTimestamp:2026-09-17 17:31:02.50830139 +0000 UTC m=+0.398063149,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.519729 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.519715 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 17:31:02.519788 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.519737 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 17:31:02.519852 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.519812 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:02.520742 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.520720 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 17:31:02.520742 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.520722 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 17:31:02.520742 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.520745 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 17:31:02.520919 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.520837 2566 reconstruct.go:97] "Volume reconstruction finished" Sep 17 17:31:02.520919 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.520847 2566 reconciler.go:29] "Reconciler: start to sync state" Sep 17 17:31:02.521476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521463 2566 factory.go:55] Registering systemd factory Sep 17 17:31:02.521517 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521479 2566 factory.go:223] Registration of the systemd container factory successfully Sep 17 17:31:02.521678 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521666 2566 factory.go:153] Registering CRI-O factory Sep 17 17:31:02.521720 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521681 2566 factory.go:223] Registration of the crio container factory successfully Sep 17 17:31:02.521749 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521725 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 17:31:02.521749 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521743 2566 factory.go:103] Registering Raw factory Sep 17 17:31:02.521804 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.521753 2566 manager.go:1196] Started watching for new ooms in manager Sep 17 17:31:02.522050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.522041 2566 manager.go:349] Starting recovery of all containers Sep 17 17:31:02.533255 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.533236 2566 manager.go:354] Recovery completed Sep 17 17:31:02.533851 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.533819 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 17:31:02.533947 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.533907 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 17:31:02.538599 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.538586 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.541172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541151 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.541264 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541191 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.541264 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541207 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.541893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541878 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 17:31:02.541893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541892 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 17:31:02.542022 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541896 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.542022 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.541910 2566 state_mem.go:36] "Initialized new in-memory state store" Sep 17 17:31:02.542598 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.542580 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.542660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.542606 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.542660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.542616 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.543980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.543964 2566 policy_none.go:49] "None policy: Start" Sep 17 17:31:02.544047 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.543983 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 17:31:02.544047 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.543997 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 17:31:02.545822 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.545808 2566 policy_none.go:47] "Start" Sep 17 17:31:02.553104 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.553038 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd5447d61 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.541172065 +0000 UTC m=+0.430933827,LastTimestamp:2026-09-17 17:31:02.541172065 +0000 UTC m=+0.430933827,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.562685 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.562618 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd544e1a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-117.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.541197731 +0000 UTC m=+0.430959492,LastTimestamp:2026-09-17 17:31:02.541197731 +0000 UTC m=+0.430959492,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.587969 2566 manager.go:337] "Starting Device Plugin manager" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.588008 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.588021 2566 server.go:87] "Starting device plugin registration server" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.588307 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.588319 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.588412 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.588494 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.588501 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.588805 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd5451d07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-117.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.541212935 +0000 UTC m=+0.430974697,LastTimestamp:2026-09-17 17:31:02.541212935 +0000 UTC m=+0.430974697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.589129 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 17:31:02.592546 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.589181 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:02.601433 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.601341 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-117.ec2.internal.18d62bfbd5447d61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd5447d61 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.541172065 +0000 UTC m=+0.430933827,LastTimestamp:2026-09-17 17:31:02.542596082 +0000 UTC m=+0.432357842,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.603871 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.603854 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t2j7q" Sep 17 17:31:02.610168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.610151 2566 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2602 into the system container \"/system.slice\": " Sep 17 17:31:02.612324 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.612251 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-117.ec2.internal.18d62bfbd544e1a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd544e1a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-117.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.541197731 +0000 UTC m=+0.430959492,LastTimestamp:2026-09-17 17:31:02.542611252 +0000 UTC m=+0.432373012,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.618998 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.618981 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t2j7q" Sep 17 17:31:02.623273 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.623209 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-117.ec2.internal.18d62bfbd5451d07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-117.ec2.internal.18d62bfbd5451d07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-117.ec2.internal,UID:ip-10-0-142-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-117.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-117.ec2.internal,},FirstTimestamp:2026-09-17 17:31:02.541212935 +0000 UTC m=+0.430974697,LastTimestamp:2026-09-17 17:31:02.542619907 +0000 UTC m=+0.432381667,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-117.ec2.internal,}" Sep 17 17:31:02.689265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.689185 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.690293 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.690273 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.690398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.690311 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.690398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.690323 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.690398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.690377 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.701071 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.701052 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.701165 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.701076 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-117.ec2.internal\": node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:02.733140 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.732963 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:02.814298 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.814260 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 17:31:02.815512 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.815486 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 17:31:02.815610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.815551 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 17:31:02.815610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.815580 2566 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 17:31:02.815693 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.815628 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 17:31:02.817755 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.817734 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:31:02.833825 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.833808 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:02.916056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.916003 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal"] Sep 17 17:31:02.916162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.916110 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.917564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.917541 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.917681 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.917572 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.917681 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.917583 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.919774 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.919762 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.919898 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.919882 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.919968 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.919910 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.920563 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.920547 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.920653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.920577 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.920653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.920590 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.920653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.920553 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.920653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.920640 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.920653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.920650 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.922705 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.922687 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.922785 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.922716 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:02.922838 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.922794 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2caaeb284115359112fb85833e279785-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal\" (UID: \"2caaeb284115359112fb85833e279785\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.922838 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.922822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2caaeb284115359112fb85833e279785-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal\" (UID: \"2caaeb284115359112fb85833e279785\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.922926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.922850 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf78f1345f70a888653325ba64aacb3d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-117.ec2.internal\" (UID: \"bf78f1345f70a888653325ba64aacb3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.923452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.923436 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:02.923536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.923459 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:02.923536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:02.923470 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:02.934876 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.934860 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:02.945062 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.945022 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-117.ec2.internal\" not found" node="ip-10-0-142-117.ec2.internal" Sep 17 17:31:02.949602 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:02.949587 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-117.ec2.internal\" not found" node="ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.023794 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.023766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2caaeb284115359112fb85833e279785-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal\" (UID: \"2caaeb284115359112fb85833e279785\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.023794 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.023793 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2caaeb284115359112fb85833e279785-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal\" (UID: \"2caaeb284115359112fb85833e279785\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.023986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.023814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf78f1345f70a888653325ba64aacb3d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-117.ec2.internal\" (UID: \"bf78f1345f70a888653325ba64aacb3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.023986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.023858 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf78f1345f70a888653325ba64aacb3d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-117.ec2.internal\" (UID: \"bf78f1345f70a888653325ba64aacb3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.023986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.023880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2caaeb284115359112fb85833e279785-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal\" (UID: \"2caaeb284115359112fb85833e279785\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.023986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.023890 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2caaeb284115359112fb85833e279785-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal\" (UID: \"2caaeb284115359112fb85833e279785\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.035902 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:03.035873 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:03.136615 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:03.136592 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:03.237097 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:03.237038 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:03.247192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.247174 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:03.253307 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:03.253292 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" Sep 17 17:31:04.665163 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:04.665145 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:04.765713 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:04.765678 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:04.772836 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.772820 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 17:31:04.772984 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.772945 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 17:31:04.865865 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:04.865810 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:04.902229 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.902211 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:31:04.952601 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.952548 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:26:03 +0000 UTC" deadline="2028-06-04 19:39:16.578521456 +0000 UTC" Sep 17 17:31:04.952601 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.952593 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15026h8m11.625931909s" Sep 17 17:31:04.963859 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.963842 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:31:04.964906 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.964890 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:31:04.964982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.964919 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:31:04.964982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:04.964928 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:31:04.965977 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:04.965948 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:05.066192 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:05.066155 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:05.166844 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:05.166780 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-117.ec2.internal\" not found" Sep 17 17:31:05.230737 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.230717 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:31:05.233404 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:05.233370 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caaeb284115359112fb85833e279785.slice/crio-39d203e84c2b091d4a7fb18669be109ea4d6e534eb76506127d11f5dc3550252 WatchSource:0}: Error finding container 39d203e84c2b091d4a7fb18669be109ea4d6e534eb76506127d11f5dc3550252: Status 404 returned error can't find the container with id 39d203e84c2b091d4a7fb18669be109ea4d6e534eb76506127d11f5dc3550252 Sep 17 17:31:05.233841 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:05.233821 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf78f1345f70a888653325ba64aacb3d.slice/crio-b47fefe5835668d864f15e55bf8c498c68725140228b633169039d647e6eac98 WatchSource:0}: Error finding container b47fefe5835668d864f15e55bf8c498c68725140228b633169039d647e6eac98: Status 404 returned error can't find the container with id b47fefe5835668d864f15e55bf8c498c68725140228b633169039d647e6eac98 Sep 17 17:31:05.238561 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.238544 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:31:05.248195 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.248170 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" Sep 17 17:31:05.262152 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.262130 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 17:31:05.263634 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.263622 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" Sep 17 17:31:05.276291 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.276264 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:31:05.280745 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.280732 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 17:31:05.707613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.707584 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:31:05.825590 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.825560 2566 apiserver.go:52] "Watching apiserver" Sep 17 17:31:05.832555 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.832537 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 17:31:05.833580 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.833560 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk","openshift-cluster-node-tuning-operator/tuned-2fjpn","openshift-dns/node-resolver-vppd9","openshift-image-registry/node-ca-7g2lv","openshift-multus/multus-additional-cni-plugins-q85q4","openshift-network-operator/iptables-alerter-5587h","kube-system/konnectivity-agent-t28t7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal","openshift-multus/multus-s9jkx","openshift-multus/network-metrics-daemon-lbj98","openshift-network-diagnostics/network-check-target-lcqhc","openshift-ovn-kubernetes/ovnkube-node-5zwsd","kube-system/global-pull-secret-syncer-qvfw4"] Sep 17 17:31:05.837261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.837245 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.839485 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.839465 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.842785 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.842766 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.843137 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.843118 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.844157 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.844143 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v8vpq\"" Sep 17 17:31:05.844258 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.844245 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.844346 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.844332 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 17:31:05.844966 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.844935 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.845055 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.844941 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tkjl5\"" Sep 17 17:31:05.845378 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.845360 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 17:31:05.845472 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.845426 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.845472 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.845366 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.845576 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.845545 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 17:31:05.845852 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.845838 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 17:31:05.846389 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.846373 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r767r\"" Sep 17 17:31:05.846597 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.846578 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.846865 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.846849 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 17:31:05.847278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.847260 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.847837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.847820 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 17:31:05.847923 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.847890 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.847998 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.847923 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hk6h6\"" Sep 17 17:31:05.849675 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.849658 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.850331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.850312 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.850490 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.850468 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 17:31:05.850640 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.850624 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.850713 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.850672 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5m2wp\"" Sep 17 17:31:05.850770 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.850725 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.851924 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.851897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:05.852078 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:05.852017 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:05.854254 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.854236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:05.856679 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.856656 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.859875 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.859849 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-log-socket\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.859897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.860012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.859935 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hl5m\" (UniqueName: \"kubernetes.io/projected/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-kube-api-access-9hl5m\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.860120 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82eb104a-31c2-4105-9f71-8618de3fa675-host-slash\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.860120 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-kubelet\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860190 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-ovn\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860190 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-run-ovn-kubernetes\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860255 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysconfig\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.860255 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860239 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-sys\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.860338 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-node-log\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860338 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovnkube-script-lib\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860338 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysctl-d\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.860476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860356 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr84\" (UniqueName: \"kubernetes.io/projected/6139ea1f-0518-4aa3-a711-5df79187101b-kube-api-access-vsr84\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.860476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb353df6-c110-4406-9efb-e29e653be6dc-hosts-file\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.860476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-host\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.860476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860421 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cnibin\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.860476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860440 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-slash\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860458 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-var-lib-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-var-lib-kubelet\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.860704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6139ea1f-0518-4aa3-a711-5df79187101b-tmp\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.860704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860520 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb353df6-c110-4406-9efb-e29e653be6dc-tmp-dir\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.860704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860545 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82eb104a-31c2-4105-9f71-8618de3fa675-iptables-alerter-script\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.860704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860588 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-systemd-units\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.860704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860628 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-systemd\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tp7\" (UniqueName: \"kubernetes.io/projected/eb353df6-c110-4406-9efb-e29e653be6dc-kube-api-access-x7tp7\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860830 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860886 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h848r\" (UniqueName: \"kubernetes.io/projected/82eb104a-31c2-4105-9f71-8618de3fa675-kube-api-access-h848r\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.860919 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-run-netns\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861014 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861048 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-env-overrides\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-run\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6139ea1f-0518-4aa3-a711-5df79187101b-etc-tuned\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861242 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-cni-bin\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861388 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-cni-netd\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovnkube-config\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861456 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-kubernetes\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861499 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysctl-conf\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-host\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861558 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtsn\" (UniqueName: \"kubernetes.io/projected/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-kube-api-access-rrtsn\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861602 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-system-cni-dir\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861631 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-systemd\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovn-node-metrics-cert\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqcb\" (UniqueName: \"kubernetes.io/projected/e2a534b1-b148-4838-8aef-4fd9c9f5414a-kube-api-access-mxqcb\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861718 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-modprobe-d\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861751 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-lib-modules\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861819 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-serviceca\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861876 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.861929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861915 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.862341 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.861985 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-etc-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.863762 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.863744 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:05.863848 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:05.863820 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:05.865300 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.865211 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.865300 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.865277 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qbmlk\"" Sep 17 17:31:05.865531 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.865282 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 17:31:05.865596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.865563 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 17:31:05.865948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.865929 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 17:31:05.865948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.865937 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.866187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866059 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.866187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866174 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kjffg\"" Sep 17 17:31:05.866187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866185 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bjgfm\"" Sep 17 17:31:05.866354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866280 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 17:31:05.866406 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866377 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 17:31:05.866539 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866496 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 17:31:05.866691 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866540 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 17:31:05.866691 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866557 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 17:31:05.866691 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866588 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:05.866691 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.866669 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bm94f\"" Sep 17 17:31:05.866946 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:05.866921 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:05.867613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.867595 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 17:31:05.883666 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.883642 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 17:31:05.933429 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.933405 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9wplj" Sep 17 17:31:05.945119 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.945098 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9wplj" Sep 17 17:31:05.949404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.949389 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 17:31:05.963122 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963063 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-etc-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963122 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963095 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.963261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hl5m\" (UniqueName: \"kubernetes.io/projected/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-kube-api-access-9hl5m\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.963261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963151 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d889d683-785d-40d7-a042-9e95c805b529-kubelet-config\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:05.963261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d360eb6-6699-4278-95fc-55d7f937ff4c-konnectivity-ca\") pod \"konnectivity-agent-t28t7\" (UID: \"6d360eb6-6699-4278-95fc-55d7f937ff4c\") " pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:05.963261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963184 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-etc-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-sys\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.963261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963239 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb353df6-c110-4406-9efb-e29e653be6dc-hosts-file\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963275 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-sys\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963323 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-host\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963378 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb353df6-c110-4406-9efb-e29e653be6dc-hosts-file\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5k7p\" (UniqueName: \"kubernetes.io/projected/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-kube-api-access-c5k7p\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963400 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-host\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-socket-dir-parent\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963493 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-k8s-cni-cncf-io\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.963541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963515 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-conf-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963537 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovnkube-script-lib\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysctl-d\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963610 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-cni-multus\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963645 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-slash\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-var-lib-kubelet\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb353df6-c110-4406-9efb-e29e653be6dc-tmp-dir\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-sys-fs\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysctl-d\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963731 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-slash\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82eb104a-31c2-4105-9f71-8618de3fa675-iptables-alerter-script\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963770 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-var-lib-kubelet\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963719 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963838 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tp7\" (UniqueName: \"kubernetes.io/projected/eb353df6-c110-4406-9efb-e29e653be6dc-kube-api-access-x7tp7\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963843 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.963901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963900 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.963929 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-run-netns\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-env-overrides\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-run\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6139ea1f-0518-4aa3-a711-5df79187101b-etc-tuned\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb353df6-c110-4406-9efb-e29e653be6dc-tmp-dir\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-run-netns\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-cni-bin\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964203 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-run\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-host\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovnkube-script-lib\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964346 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-host\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82eb104a-31c2-4105-9f71-8618de3fa675-iptables-alerter-script\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-system-cni-dir\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964370 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-cni-bin\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964401 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-system-cni-dir\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.964632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964454 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-device-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964474 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mm6b\" (UniqueName: \"kubernetes.io/projected/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-kube-api-access-7mm6b\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-systemd\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-systemd\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-env-overrides\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964604 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-serviceca\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-log-socket\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d889d683-785d-40d7-a042-9e95c805b529-dbus\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82eb104a-31c2-4105-9f71-8618de3fa675-host-slash\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-kubelet\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-ovn\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-run-ovn-kubernetes\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964854 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964873 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysconfig\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr84\" (UniqueName: \"kubernetes.io/projected/6139ea1f-0518-4aa3-a711-5df79187101b-kube-api-access-vsr84\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.965502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964908 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82eb104a-31c2-4105-9f71-8618de3fa675-host-slash\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-node-log\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964923 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964948 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-log-socket\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cnibin\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.964989 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-node-log\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965024 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cnibin\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965027 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-kubelet\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965059 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-run-ovn-kubernetes\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965068 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-serviceca\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965075 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysconfig\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965085 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-registration-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965109 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-run-ovn\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-cni-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-netns\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965196 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.966323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965200 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-cni-bin\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965221 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-daemon-config\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-var-lib-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965286 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6139ea1f-0518-4aa3-a711-5df79187101b-tmp\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-var-lib-openvswitch\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-socket-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-system-cni-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965370 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-cnibin\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965413 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmnw\" (UniqueName: \"kubernetes.io/projected/6cf9219f-4e4a-482d-a394-b235e5723aed-kube-api-access-brmnw\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965445 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-systemd-units\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965469 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-systemd\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965519 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-multus-certs\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965547 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h848r\" (UniqueName: \"kubernetes.io/projected/82eb104a-31c2-4105-9f71-8618de3fa675-kube-api-access-h848r\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-systemd\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965567 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-systemd-units\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.966904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965624 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-kubelet\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-cni-netd\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965720 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovnkube-config\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965741 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2a534b1-b148-4838-8aef-4fd9c9f5414a-host-cni-netd\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-kubernetes\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965772 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysctl-conf\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965796 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtsn\" (UniqueName: \"kubernetes.io/projected/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-kube-api-access-rrtsn\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965811 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-kubernetes\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965841 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d360eb6-6699-4278-95fc-55d7f937ff4c-agent-certs\") pod \"konnectivity-agent-t28t7\" (UID: \"6d360eb6-6699-4278-95fc-55d7f937ff4c\") " pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cf9219f-4e4a-482d-a394-b235e5723aed-cni-binary-copy\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965889 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-etc-kubernetes\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovn-node-metrics-cert\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-sysctl-conf\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.965981 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqcb\" (UniqueName: \"kubernetes.io/projected/e2a534b1-b148-4838-8aef-4fd9c9f5414a-kube-api-access-mxqcb\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.967567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.966007 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-modprobe-d\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.966110 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-etc-modprobe-d\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.966191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-lib-modules\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.966213 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovnkube-config\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.966221 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-hostroot\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.966348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6139ea1f-0518-4aa3-a711-5df79187101b-lib-modules\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.968128 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6139ea1f-0518-4aa3-a711-5df79187101b-tmp\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.968239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.968161 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6139ea1f-0518-4aa3-a711-5df79187101b-etc-tuned\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.968527 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.968421 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2a534b1-b148-4838-8aef-4fd9c9f5414a-ovn-node-metrics-cert\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.979533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.979513 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tp7\" (UniqueName: \"kubernetes.io/projected/eb353df6-c110-4406-9efb-e29e653be6dc-kube-api-access-x7tp7\") pod \"node-resolver-vppd9\" (UID: \"eb353df6-c110-4406-9efb-e29e653be6dc\") " pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:05.980369 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.980351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hl5m\" (UniqueName: \"kubernetes.io/projected/d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe-kube-api-access-9hl5m\") pod \"multus-additional-cni-plugins-q85q4\" (UID: \"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe\") " pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:05.988898 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.988855 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr84\" (UniqueName: \"kubernetes.io/projected/6139ea1f-0518-4aa3-a711-5df79187101b-kube-api-access-vsr84\") pod \"tuned-2fjpn\" (UID: \"6139ea1f-0518-4aa3-a711-5df79187101b\") " pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:05.989281 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.989257 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h848r\" (UniqueName: \"kubernetes.io/projected/82eb104a-31c2-4105-9f71-8618de3fa675-kube-api-access-h848r\") pod \"iptables-alerter-5587h\" (UID: \"82eb104a-31c2-4105-9f71-8618de3fa675\") " pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:05.990190 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.990171 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqcb\" (UniqueName: \"kubernetes.io/projected/e2a534b1-b148-4838-8aef-4fd9c9f5414a-kube-api-access-mxqcb\") pod \"ovnkube-node-5zwsd\" (UID: \"e2a534b1-b148-4838-8aef-4fd9c9f5414a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:05.990793 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:05.990776 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtsn\" (UniqueName: \"kubernetes.io/projected/2b69f4eb-f959-4d3c-9331-3a8ddf0af363-kube-api-access-rrtsn\") pod \"node-ca-7g2lv\" (UID: \"2b69f4eb-f959-4d3c-9331-3a8ddf0af363\") " pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:06.067090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-daemon-config\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067089 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-socket-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-system-cni-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-cnibin\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmnw\" (UniqueName: \"kubernetes.io/projected/6cf9219f-4e4a-482d-a394-b235e5723aed-kube-api-access-brmnw\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067195 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-multus-certs\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067223 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-system-cni-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-socket-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-kubelet\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067315 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.067317 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067346 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d360eb6-6699-4278-95fc-55d7f937ff4c-agent-certs\") pod \"konnectivity-agent-t28t7\" (UID: \"6d360eb6-6699-4278-95fc-55d7f937ff4c\") " pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cf9219f-4e4a-482d-a394-b235e5723aed-cni-binary-copy\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.067441 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:06.567387353 +0000 UTC m=+3.129552224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067461 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-etc-kubernetes\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-hostroot\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067540 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d889d683-785d-40d7-a042-9e95c805b529-kubelet-config\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067581 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-cnibin\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d360eb6-6699-4278-95fc-55d7f937ff4c-konnectivity-ca\") pod \"konnectivity-agent-t28t7\" (UID: \"6d360eb6-6699-4278-95fc-55d7f937ff4c\") " pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5k7p\" (UniqueName: \"kubernetes.io/projected/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-kube-api-access-c5k7p\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:06.067660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067645 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-kubelet\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067647 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-socket-dir-parent\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067720 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-daemon-config\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.067727 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-socket-dir-parent\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067781 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-k8s-cni-cncf-io\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.067794 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:06.567778958 +0000 UTC m=+3.129943814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-k8s-cni-cncf-io\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067800 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-conf-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067857 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-multus-certs\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-cni-multus\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067890 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-cni-multus\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-sys-fs\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067906 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cf9219f-4e4a-482d-a394-b235e5723aed-cni-binary-copy\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067932 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-sys-fs\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.067937 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:06.068143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-device-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068005 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-conf-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068024 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mm6b\" (UniqueName: \"kubernetes.io/projected/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-kube-api-access-7mm6b\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068068 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d889d683-785d-40d7-a042-9e95c805b529-kubelet-config\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068116 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-hostroot\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d889d683-785d-40d7-a042-9e95c805b529-dbus\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068190 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-registration-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068187 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-etc-kubernetes\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-cni-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-netns\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068368 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-registration-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-cni-bin\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068483 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-device-dir\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068506 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-var-lib-cni-bin\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068509 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-host-run-netns\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068521 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d889d683-785d-40d7-a042-9e95c805b529-dbus\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:06.068780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068536 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf9219f-4e4a-482d-a394-b235e5723aed-multus-cni-dir\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.069623 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.068931 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d360eb6-6699-4278-95fc-55d7f937ff4c-konnectivity-ca\") pod \"konnectivity-agent-t28t7\" (UID: \"6d360eb6-6699-4278-95fc-55d7f937ff4c\") " pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:06.070329 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.070296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d360eb6-6699-4278-95fc-55d7f937ff4c-agent-certs\") pod \"konnectivity-agent-t28t7\" (UID: \"6d360eb6-6699-4278-95fc-55d7f937ff4c\") " pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:06.087171 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.087148 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:06.087171 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.087170 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:06.087343 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.087183 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:06.087343 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.087245 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:31:06.587227789 +0000 UTC m=+3.149392627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:06.090185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.090160 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmnw\" (UniqueName: \"kubernetes.io/projected/6cf9219f-4e4a-482d-a394-b235e5723aed-kube-api-access-brmnw\") pod \"multus-s9jkx\" (UID: \"6cf9219f-4e4a-482d-a394-b235e5723aed\") " pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.090297 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.090206 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5k7p\" (UniqueName: \"kubernetes.io/projected/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-kube-api-access-c5k7p\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:06.090297 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.090275 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mm6b\" (UniqueName: \"kubernetes.io/projected/5d8f0d7f-3691-4412-9d8c-8700b47c58d9-kube-api-access-7mm6b\") pod \"aws-ebs-csi-driver-node-qlwjk\" (UID: \"5d8f0d7f-3691-4412-9d8c-8700b47c58d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.147506 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.147481 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5587h" Sep 17 17:31:06.147844 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.147797 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" event={"ID":"bf78f1345f70a888653325ba64aacb3d","Type":"ContainerStarted","Data":"b47fefe5835668d864f15e55bf8c498c68725140228b633169039d647e6eac98"} Sep 17 17:31:06.149049 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.149024 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" event={"ID":"2caaeb284115359112fb85833e279785","Type":"ContainerStarted","Data":"39d203e84c2b091d4a7fb18669be109ea4d6e534eb76506127d11f5dc3550252"} Sep 17 17:31:06.155082 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.155059 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:06.158772 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.158743 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82eb104a_31c2_4105_9f71_8618de3fa675.slice/crio-672143e94cc9ff07edc4ab0276ba39cfb2abbf7234cc3606c89d412e73868a89 WatchSource:0}: Error finding container 672143e94cc9ff07edc4ab0276ba39cfb2abbf7234cc3606c89d412e73868a89: Status 404 returned error can't find the container with id 672143e94cc9ff07edc4ab0276ba39cfb2abbf7234cc3606c89d412e73868a89 Sep 17 17:31:06.165027 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.165003 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a534b1_b148_4838_8aef_4fd9c9f5414a.slice/crio-fe97a0ac7cd89e165ccf41607aece657ceceea6558ddb3f0dbdb5e65e25f6de8 WatchSource:0}: Error finding container fe97a0ac7cd89e165ccf41607aece657ceceea6558ddb3f0dbdb5e65e25f6de8: Status 404 returned error can't find the container with id fe97a0ac7cd89e165ccf41607aece657ceceea6558ddb3f0dbdb5e65e25f6de8 Sep 17 17:31:06.165126 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.165082 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" Sep 17 17:31:06.173331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.173314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vppd9" Sep 17 17:31:06.173635 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.173616 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6139ea1f_0518_4aa3_a711_5df79187101b.slice/crio-1dd7441227251df26d8f2560ae600f210ff9225dec71103ac79507ffb705aeda WatchSource:0}: Error finding container 1dd7441227251df26d8f2560ae600f210ff9225dec71103ac79507ffb705aeda: Status 404 returned error can't find the container with id 1dd7441227251df26d8f2560ae600f210ff9225dec71103ac79507ffb705aeda Sep 17 17:31:06.179377 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.179358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7g2lv" Sep 17 17:31:06.181538 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.181518 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb353df6_c110_4406_9efb_e29e653be6dc.slice/crio-03643fdb79b820cf0cc32ba32e365afb7dab414a8fa71027b58b64683e2316d4 WatchSource:0}: Error finding container 03643fdb79b820cf0cc32ba32e365afb7dab414a8fa71027b58b64683e2316d4: Status 404 returned error can't find the container with id 03643fdb79b820cf0cc32ba32e365afb7dab414a8fa71027b58b64683e2316d4 Sep 17 17:31:06.186865 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.186848 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q85q4" Sep 17 17:31:06.187504 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.187485 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b69f4eb_f959_4d3c_9331_3a8ddf0af363.slice/crio-50a1a5aa6f43f61eb3db211db4c9d66e491974e5001cc7415b6d77f6f5896b01 WatchSource:0}: Error finding container 50a1a5aa6f43f61eb3db211db4c9d66e491974e5001cc7415b6d77f6f5896b01: Status 404 returned error can't find the container with id 50a1a5aa6f43f61eb3db211db4c9d66e491974e5001cc7415b6d77f6f5896b01 Sep 17 17:31:06.192676 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.192662 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:06.194335 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.194317 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd317ebb7_0fa0_4b76_a2f0_97e5e0343bbe.slice/crio-3266b132d614b92574a717b9a0bb28698beff26ddc8008bf805171ce0e291264 WatchSource:0}: Error finding container 3266b132d614b92574a717b9a0bb28698beff26ddc8008bf805171ce0e291264: Status 404 returned error can't find the container with id 3266b132d614b92574a717b9a0bb28698beff26ddc8008bf805171ce0e291264 Sep 17 17:31:06.200183 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.200160 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" Sep 17 17:31:06.200521 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.200498 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d360eb6_6699_4278_95fc_55d7f937ff4c.slice/crio-4594abd7c0953b4805d4a5beb4387ca45d0600a4ac8098ba2ee77adb29784b65 WatchSource:0}: Error finding container 4594abd7c0953b4805d4a5beb4387ca45d0600a4ac8098ba2ee77adb29784b65: Status 404 returned error can't find the container with id 4594abd7c0953b4805d4a5beb4387ca45d0600a4ac8098ba2ee77adb29784b65 Sep 17 17:31:06.206055 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.206038 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s9jkx" Sep 17 17:31:06.209074 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.209053 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8f0d7f_3691_4412_9d8c_8700b47c58d9.slice/crio-8b5dfb43b30707ab84debc4fcbf636c7850d3acceffd21102f3a9ed8c5c8bbc5 WatchSource:0}: Error finding container 8b5dfb43b30707ab84debc4fcbf636c7850d3acceffd21102f3a9ed8c5c8bbc5: Status 404 returned error can't find the container with id 8b5dfb43b30707ab84debc4fcbf636c7850d3acceffd21102f3a9ed8c5c8bbc5 Sep 17 17:31:06.214424 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:06.214405 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf9219f_4e4a_482d_a394_b235e5723aed.slice/crio-738160a9bdd79ee73d36befe659481ab5ed1bad10ac92a5137c7a9cc29b891c5 WatchSource:0}: Error finding container 738160a9bdd79ee73d36befe659481ab5ed1bad10ac92a5137c7a9cc29b891c5: Status 404 returned error can't find the container with id 738160a9bdd79ee73d36befe659481ab5ed1bad10ac92a5137c7a9cc29b891c5 Sep 17 17:31:06.571393 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.571304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:06.571393 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.571351 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:06.571593 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.571483 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:06.571593 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.571484 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:06.571593 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.571550 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:07.571529994 +0000 UTC m=+4.133694848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:06.571593 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.571576 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:07.57155878 +0000 UTC m=+4.133723625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:06.671851 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.671810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:06.672035 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.671980 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:06.672035 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.672000 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:06.672035 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.672011 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:06.672208 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:06.672082 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:31:07.672063852 +0000 UTC m=+4.234228692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:06.948976 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.948875 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:26:05 +0000 UTC" deadline="2028-02-23 19:08:37.541565832 +0000 UTC" Sep 17 17:31:06.948976 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:06.948908 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12577h37m30.592662239s" Sep 17 17:31:07.144344 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.144310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:07.144344 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.144336 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:07.144574 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.144438 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:07.144574 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.144545 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:07.154403 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.154339 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t28t7" event={"ID":"6d360eb6-6699-4278-95fc-55d7f937ff4c","Type":"ContainerStarted","Data":"4594abd7c0953b4805d4a5beb4387ca45d0600a4ac8098ba2ee77adb29784b65"} Sep 17 17:31:07.156363 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.156292 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerStarted","Data":"3266b132d614b92574a717b9a0bb28698beff26ddc8008bf805171ce0e291264"} Sep 17 17:31:07.157787 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.157714 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vppd9" event={"ID":"eb353df6-c110-4406-9efb-e29e653be6dc","Type":"ContainerStarted","Data":"03643fdb79b820cf0cc32ba32e365afb7dab414a8fa71027b58b64683e2316d4"} Sep 17 17:31:07.159422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.159397 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5587h" event={"ID":"82eb104a-31c2-4105-9f71-8618de3fa675","Type":"ContainerStarted","Data":"672143e94cc9ff07edc4ab0276ba39cfb2abbf7234cc3606c89d412e73868a89"} Sep 17 17:31:07.162216 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.162130 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" event={"ID":"5d8f0d7f-3691-4412-9d8c-8700b47c58d9","Type":"ContainerStarted","Data":"8b5dfb43b30707ab84debc4fcbf636c7850d3acceffd21102f3a9ed8c5c8bbc5"} Sep 17 17:31:07.163790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.163727 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7g2lv" event={"ID":"2b69f4eb-f959-4d3c-9331-3a8ddf0af363","Type":"ContainerStarted","Data":"50a1a5aa6f43f61eb3db211db4c9d66e491974e5001cc7415b6d77f6f5896b01"} Sep 17 17:31:07.165100 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.165016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" event={"ID":"6139ea1f-0518-4aa3-a711-5df79187101b","Type":"ContainerStarted","Data":"1dd7441227251df26d8f2560ae600f210ff9225dec71103ac79507ffb705aeda"} Sep 17 17:31:07.166670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.166560 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"fe97a0ac7cd89e165ccf41607aece657ceceea6558ddb3f0dbdb5e65e25f6de8"} Sep 17 17:31:07.168609 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.168584 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s9jkx" event={"ID":"6cf9219f-4e4a-482d-a394-b235e5723aed","Type":"ContainerStarted","Data":"738160a9bdd79ee73d36befe659481ab5ed1bad10ac92a5137c7a9cc29b891c5"} Sep 17 17:31:07.579581 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.579544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:07.579769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.579596 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:07.579841 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.579787 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:07.579890 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.579847 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:09.57982835 +0000 UTC m=+6.141993193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:07.580284 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.580265 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:07.580364 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.580321 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:09.580304788 +0000 UTC m=+6.142469631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:07.680293 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.680257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:07.680454 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.680442 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:07.680511 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.680467 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:07.680511 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.680479 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:07.680627 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:07.680539 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:31:09.68051972 +0000 UTC m=+6.242684572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:07.949732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.949646 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:26:05 +0000 UTC" deadline="2028-02-15 22:39:50.986777402 +0000 UTC" Sep 17 17:31:07.949732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:07.949686 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12389h8m43.037096592s" Sep 17 17:31:08.147983 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:08.147940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:08.148197 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:08.148120 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:09.143658 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:09.143586 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:09.144176 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.143712 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:09.144176 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:09.144110 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:09.144304 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.144205 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:09.599693 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:09.599655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:09.599693 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:09.599702 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:09.599920 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.599840 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:09.599920 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.599895 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:13.599879829 +0000 UTC m=+10.162044668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:09.600352 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.600292 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:09.600352 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.600353 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:13.600337274 +0000 UTC m=+10.162502116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:09.700821 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:09.700785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:09.700995 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.700964 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:09.700995 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.700984 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:09.700995 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.700996 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:09.701171 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:09.701055 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:31:13.701036695 +0000 UTC m=+10.263201546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:10.144645 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:10.144614 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:10.145109 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:10.144725 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:11.144233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:11.144178 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:11.144387 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:11.144305 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:11.144387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:11.144359 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:11.144511 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:11.144482 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:12.143474 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:12.143442 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:12.143838 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:12.143595 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:13.144388 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:13.143887 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:13.144388 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:13.143935 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:13.144388 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.144047 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:13.144388 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.144154 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:13.637172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:13.636420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:13.637172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:13.636467 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:13.637172 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.636639 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:13.637172 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.636701 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:21.636682654 +0000 UTC m=+18.198847506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:13.637172 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.637089 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:13.637172 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.637140 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:21.637124691 +0000 UTC m=+18.199289554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:13.738826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:13.737301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:13.738826 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.737555 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:13.738826 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.737576 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:13.738826 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.737590 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:13.738826 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:13.737653 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:31:21.737635295 +0000 UTC m=+18.299800147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:14.145351 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:14.144861 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:14.145351 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:14.145009 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:15.143777 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:15.143661 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:15.143968 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:15.143788 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:15.144266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:15.144115 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:15.144266 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:15.144230 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:16.144263 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:16.144228 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:16.144724 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:16.144377 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:17.143510 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:17.143481 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:17.143677 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:17.143576 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:17.143677 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:17.143602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:17.143677 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:17.143669 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:18.144425 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:18.144381 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:18.144839 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:18.144540 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:19.143376 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:19.143343 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:19.143576 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:19.143345 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:19.143576 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:19.143484 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:19.143576 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:19.143548 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:20.143905 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:20.143868 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:20.144352 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:20.144027 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:21.144133 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:21.144061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:21.144558 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.144175 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:21.144558 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:21.144229 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:21.144558 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.144344 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:21.692266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:21.692225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:21.692507 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:21.692274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:21.692507 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.692389 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:21.692507 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.692453 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:37.692435464 +0000 UTC m=+34.254600302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:21.692507 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.692398 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:21.692677 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.692533 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:37.69251697 +0000 UTC m=+34.254681811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:21.793232 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:21.793193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:21.793399 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.793365 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:21.793399 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.793389 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:21.793399 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.793399 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:21.793539 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:21.793451 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:31:37.793435644 +0000 UTC m=+34.355600482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:22.143876 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:22.143722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:22.144057 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:22.143869 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:23.143988 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:23.143773 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:23.144405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:23.143810 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:23.144405 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:23.144075 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:23.144405 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:23.144154 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:24.144146 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.144116 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:24.144580 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:24.144264 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:24.206281 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.205994 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" event={"ID":"bf78f1345f70a888653325ba64aacb3d","Type":"ContainerStarted","Data":"0c1991ad3e041bb119e2aebffa3fdfd7ae58f6218f0533b1d7c90a543331b40c"} Sep 17 17:31:24.211178 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.211118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" event={"ID":"6139ea1f-0518-4aa3-a711-5df79187101b","Type":"ContainerStarted","Data":"29a99cd8796d75f169837394744f90bfa186470e4526dfe4d838237ca98c7fc0"} Sep 17 17:31:24.214776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.214717 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"a5ea928dccbf0293d503853e308314b9ba4cabcfa848a51314c3f0e5241b2a57"} Sep 17 17:31:24.220680 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.220562 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s9jkx" event={"ID":"6cf9219f-4e4a-482d-a394-b235e5723aed","Type":"ContainerStarted","Data":"d1fd599079cc642e00fd0490c52bd0427c034ed161b537fb0a2253d14cd2cff0"} Sep 17 17:31:24.222200 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.222144 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-117.ec2.internal" podStartSLOduration=19.222126139 podStartE2EDuration="19.222126139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:24.221678678 +0000 UTC m=+20.783843540" watchObservedRunningTime="2026-09-17 17:31:24.222126139 +0000 UTC m=+20.784291000" Sep 17 17:31:24.250278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.248136 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s9jkx" podStartSLOduration=2.513148405 podStartE2EDuration="20.248114975s" totalImagesPullingTime="17.73496657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:24.245867871 +0000 UTC m=+20.808032733" watchObservedRunningTime="2026-09-17 17:31:24.248114975 +0000 UTC m=+20.810279837" Sep 17 17:31:24.266017 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:24.265921 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2fjpn" podStartSLOduration=2.49093111 podStartE2EDuration="20.265899742s" totalImagesPullingTime="17.774968632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:24.265110793 +0000 UTC m=+20.827275654" watchObservedRunningTime="2026-09-17 17:31:24.265899742 +0000 UTC m=+20.828064618" Sep 17 17:31:25.143455 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.143414 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:25.143606 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.143418 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:25.143606 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:25.143596 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:25.143779 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:25.143743 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:25.224896 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.224820 2566 generic.go:356] "Generic (PLEG): container finished" podID="d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe" containerID="79e485679f5882951eb79e6768a9da0439a4e94d2e544e99e96260eba0cfd4ac" exitCode=0 Sep 17 17:31:25.225806 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.224904 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerDied","Data":"79e485679f5882951eb79e6768a9da0439a4e94d2e544e99e96260eba0cfd4ac"} Sep 17 17:31:25.229102 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.229078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vppd9" event={"ID":"eb353df6-c110-4406-9efb-e29e653be6dc","Type":"ContainerStarted","Data":"a390be2bd1c4e9c843a53d8c72dd6b93e52c9bfd12225d1b95b557a7199a5bb6"} Sep 17 17:31:25.230550 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.230526 2566 generic.go:356] "Generic (PLEG): container finished" podID="2caaeb284115359112fb85833e279785" containerID="d9149292fc53233c4d47a9b1cfff58b22d9cd8966a7637e78cda3dc1190b291d" exitCode=0 Sep 17 17:31:25.230667 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.230604 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" event={"ID":"2caaeb284115359112fb85833e279785","Type":"ContainerDied","Data":"d9149292fc53233c4d47a9b1cfff58b22d9cd8966a7637e78cda3dc1190b291d"} Sep 17 17:31:25.232262 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.232244 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" event={"ID":"5d8f0d7f-3691-4412-9d8c-8700b47c58d9","Type":"ContainerStarted","Data":"84c99271c724f6cf44c89151cfeae314266cd97f9648ede2bec8a900c2eae9e5"} Sep 17 17:31:25.233701 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.233682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7g2lv" event={"ID":"2b69f4eb-f959-4d3c-9331-3a8ddf0af363","Type":"ContainerStarted","Data":"3f5d142b74f5240f887c8571a9506b5b56602d2eea7af418209a5f4192d75656"} Sep 17 17:31:25.237172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.237155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"8654b255e656485ef018edbcd61f8f811c607a14e2a21ffc0a831b70aa61259d"} Sep 17 17:31:25.237256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.237213 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"8d7afa4ab05f9d265e906da86446ab858d888c1e69949a8bede0a321dd2320e1"} Sep 17 17:31:25.237312 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.237260 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"9d4b6d39c11f4233b45158b00c594dcd20a9fd6a0e041652212727faca719022"} Sep 17 17:31:25.237312 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.237289 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"dad6c9cacae7c4ec614d3442b4d0f89a44858ebfe3fb70d5695687740a1fbd69"} Sep 17 17:31:25.237410 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.237316 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"51d0d30d0c9d9768885718b2ffe67ae737e838274ce5f6e7d433a3024051665a"} Sep 17 17:31:25.238495 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.238475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t28t7" event={"ID":"6d360eb6-6699-4278-95fc-55d7f937ff4c","Type":"ContainerStarted","Data":"1c705c91192affca777de457e0060ab354cb5682d667c823c7ae1a20e38957a5"} Sep 17 17:31:25.271436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.271209 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7g2lv" podStartSLOduration=3.512547661 podStartE2EDuration="21.271192871s" totalImagesPullingTime="17.75864521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:25.27077605 +0000 UTC m=+21.832940911" watchObservedRunningTime="2026-09-17 17:31:25.271192871 +0000 UTC m=+21.833357732" Sep 17 17:31:25.289056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.288920 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t28t7" podStartSLOduration=8.117025815 podStartE2EDuration="21.288906078s" totalImagesPullingTime="13.171880263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:25.288730033 +0000 UTC m=+21.850894893" watchObservedRunningTime="2026-09-17 17:31:25.288906078 +0000 UTC m=+21.851070935" Sep 17 17:31:25.380587 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.380568 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 17:31:25.615648 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.615529 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:25.616166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.616149 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:25.642108 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.642065 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vppd9" podStartSLOduration=3.8748127820000002 podStartE2EDuration="21.642050295s" totalImagesPullingTime="17.767237513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:25.325776872 +0000 UTC m=+21.887941734" watchObservedRunningTime="2026-09-17 17:31:25.642050295 +0000 UTC m=+22.204215176" Sep 17 17:31:25.928303 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.928101 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:31:25.380583995Z","UUID":"ba79107e-e735-4725-90bf-c9713ecd2200","Handler":null,"Name":"","Endpoint":""} Sep 17 17:31:25.929843 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.929817 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 17:31:25.929843 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:25.929849 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 17:31:26.143796 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.143765 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:26.143982 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:26.143906 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:26.242324 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.242229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5587h" event={"ID":"82eb104a-31c2-4105-9f71-8618de3fa675","Type":"ContainerStarted","Data":"369486b74265550c05457a6cf27744b2683bb4a6de991d8018348d21d072d88a"} Sep 17 17:31:26.244804 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.244778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" event={"ID":"2caaeb284115359112fb85833e279785","Type":"ContainerStarted","Data":"ee131bcb86cc29898da365c8862f460dd3d0a1c8ec1de3275e58d9fff766f522"} Sep 17 17:31:26.246845 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.246713 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" event={"ID":"5d8f0d7f-3691-4412-9d8c-8700b47c58d9","Type":"ContainerStarted","Data":"61ead67a829fa239056f3523d6ec1814aed79f9d22224e32e32e7ed27b42b1b0"} Sep 17 17:31:26.247001 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.246983 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:26.247730 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.247699 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t28t7" Sep 17 17:31:26.300843 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.300801 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5587h" podStartSLOduration=4.51391916 podStartE2EDuration="22.300785461s" totalImagesPullingTime="17.786866301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:26.273212701 +0000 UTC m=+22.835377561" watchObservedRunningTime="2026-09-17 17:31:26.300785461 +0000 UTC m=+22.862950320" Sep 17 17:31:26.320369 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:26.320327 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-117.ec2.internal" podStartSLOduration=21.320313544 podStartE2EDuration="21.320313544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:26.319113173 +0000 UTC m=+22.881278033" watchObservedRunningTime="2026-09-17 17:31:26.320313544 +0000 UTC m=+22.882478400" Sep 17 17:31:27.144404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:27.144366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:27.144585 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:27.144387 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:27.144585 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:27.144489 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:27.144585 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:27.144534 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:27.251472 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:27.251426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" event={"ID":"5d8f0d7f-3691-4412-9d8c-8700b47c58d9","Type":"ContainerStarted","Data":"eef873d7d09bc9174a7cddd628f5add73c576358f381fbca2b08f7c4a34c846c"} Sep 17 17:31:27.255669 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:27.255616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"354024422a49ac9842bc4116d0e897913d4a96db7868d76a9e792761c6b05447"} Sep 17 17:31:28.144019 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:28.143991 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:28.144255 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:28.144090 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:29.144371 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.144130 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:29.144869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.144172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:29.144869 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:29.144469 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:29.144869 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:29.144540 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:29.264217 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.264176 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" event={"ID":"e2a534b1-b148-4838-8aef-4fd9c9f5414a","Type":"ContainerStarted","Data":"1972205f1182c7ce94af4da704fbb5fd4179dcd2d611b2f437b22df3b46b82c2"} Sep 17 17:31:29.265076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.265050 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:29.265256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.265162 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:29.265256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.265232 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:29.283149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.283097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:29.284000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.283934 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:31:29.301850 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.301811 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" podStartSLOduration=7.443175376 podStartE2EDuration="25.301797806s" totalImagesPullingTime="17.85862243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:29.301384035 +0000 UTC m=+25.863548896" watchObservedRunningTime="2026-09-17 17:31:29.301797806 +0000 UTC m=+25.863962666" Sep 17 17:31:29.302054 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:29.302029 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlwjk" podStartSLOduration=5.550930439 podStartE2EDuration="25.302023795s" totalImagesPullingTime="19.751093356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:27.271688563 +0000 UTC m=+23.833853425" watchObservedRunningTime="2026-09-17 17:31:29.302023795 +0000 UTC m=+25.864188662" Sep 17 17:31:30.147794 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.147759 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:30.148349 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:30.147909 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:30.620149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.620116 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qvfw4"] Sep 17 17:31:30.620295 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.620220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:30.620335 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:30.620297 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:30.624356 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.624332 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lcqhc"] Sep 17 17:31:30.624454 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.624420 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:30.624503 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:30.624488 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:30.635067 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.635035 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lbj98"] Sep 17 17:31:30.635170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:30.635114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:30.635226 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:30.635185 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:31.272497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:31.272422 2566 generic.go:356] "Generic (PLEG): container finished" podID="d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe" containerID="b53ab5b415069eeaacd6fdd3de1e6f82e8053f9a6ed4eb925c637f8aae5bc1f8" exitCode=0 Sep 17 17:31:31.273072 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:31.272504 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerDied","Data":"b53ab5b415069eeaacd6fdd3de1e6f82e8053f9a6ed4eb925c637f8aae5bc1f8"} Sep 17 17:31:32.144027 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:32.143996 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:32.144180 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:32.143996 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:32.144180 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:32.144100 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:32.144261 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:32.144185 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:32.144261 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:32.144005 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:32.144335 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:32.144293 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:33.278681 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:33.278651 2566 generic.go:356] "Generic (PLEG): container finished" podID="d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe" containerID="ca83935e6c6eaa65ce6fc9f5f84d9c94480d9a6ffd59f5bf0cfaa6ea1ee3271b" exitCode=0 Sep 17 17:31:33.279143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:33.278717 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerDied","Data":"ca83935e6c6eaa65ce6fc9f5f84d9c94480d9a6ffd59f5bf0cfaa6ea1ee3271b"} Sep 17 17:31:34.144426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:34.144256 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:34.144524 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:34.144323 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:34.144608 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:34.144357 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:34.144689 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:34.144606 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:34.144689 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:34.144494 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:34.144783 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:34.144715 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:34.282880 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:34.282844 2566 generic.go:356] "Generic (PLEG): container finished" podID="d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe" containerID="c9cfe499796ff3b915843e9fbf5de84a89145a2e2fc55485dd791076f10a4345" exitCode=0 Sep 17 17:31:34.283231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:34.282891 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerDied","Data":"c9cfe499796ff3b915843e9fbf5de84a89145a2e2fc55485dd791076f10a4345"} Sep 17 17:31:36.143865 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.143835 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:36.144312 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.143874 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:36.144312 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.143842 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:36.144312 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:36.143991 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lcqhc" podUID="eb3e211e-a983-4322-940a-1962fc9705ab" Sep 17 17:31:36.144312 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:36.144142 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qvfw4" podUID="d889d683-785d-40d7-a042-9e95c805b529" Sep 17 17:31:36.144312 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:36.144234 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:31:36.749170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.749146 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-117.ec2.internal" event="NodeReady" Sep 17 17:31:36.749319 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.749274 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 17:31:36.809440 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.809395 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw"] Sep 17 17:31:36.828015 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.827982 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8669fdd6cf-jcw8m"] Sep 17 17:31:36.828408 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.828107 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:36.832060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.832034 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 17:31:36.832191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.832077 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6m79t\"" Sep 17 17:31:36.832191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.832122 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 17:31:36.832191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.832045 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 17:31:36.832573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.832555 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 17 17:31:36.846384 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.845571 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z"] Sep 17 17:31:36.846384 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.846015 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:36.848670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.848325 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 17:31:36.848670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.848384 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b7bq9\"" Sep 17 17:31:36.848670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.848329 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 17:31:36.848670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.848578 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 17:31:36.854459 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.854432 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 17:31:36.862174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.861775 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl"] Sep 17 17:31:36.862334 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.862315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:36.865113 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.865094 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 17 17:31:36.885687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.885665 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wm2jb"] Sep 17 17:31:36.885808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.885788 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:36.888350 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.888332 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 17 17:31:36.888350 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.888345 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 17 17:31:36.888565 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.888552 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 17 17:31:36.888684 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.888655 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 17 17:31:36.899242 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.899221 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f046f2d0-0152-4b1e-856b-55f4eb088650-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw\" (UID: \"f046f2d0-0152-4b1e-856b-55f4eb088650\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:36.899460 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.899435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfdn\" (UniqueName: \"kubernetes.io/projected/f046f2d0-0152-4b1e-856b-55f4eb088650-kube-api-access-pkfdn\") pod \"managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw\" (UID: \"f046f2d0-0152-4b1e-856b-55f4eb088650\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:36.906644 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.906625 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rrssb"] Sep 17 17:31:36.906748 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.906721 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:36.909042 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.909017 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fv9xq\"" Sep 17 17:31:36.909144 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.909044 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 17:31:36.909144 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.909030 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 17:31:36.922266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922236 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl"] Sep 17 17:31:36.922387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922277 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw"] Sep 17 17:31:36.922387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922293 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z"] Sep 17 17:31:36.922387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922314 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8669fdd6cf-jcw8m"] Sep 17 17:31:36.922387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922339 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrssb"] Sep 17 17:31:36.922387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922356 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wm2jb"] Sep 17 17:31:36.922629 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.922451 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:36.925646 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.925623 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mkhl7\"" Sep 17 17:31:36.925743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.925685 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 17:31:36.925743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.925708 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 17:31:36.925743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:36.925727 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 17:31:37.000810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.000810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000764 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-image-registry-private-configuration\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.000810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69f1ae7f-7b01-4378-9f18-77df3071c780-ca-trust-extracted\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.000810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-installation-pull-secrets\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000861 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-certificates\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-ca\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000935 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-bound-sa-token\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000973 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-hub\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.000999 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/984023aa-d712-4111-beda-f7c6ec7910fd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001017 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001032 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gm5f\" (UniqueName: \"kubernetes.io/projected/33f130cc-b73a-4f66-82b6-0b687aa48686-kube-api-access-5gm5f\") pod \"ingress-canary-rrssb\" (UID: \"33f130cc-b73a-4f66-82b6-0b687aa48686\") " pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001063 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bmx\" (UniqueName: \"kubernetes.io/projected/984023aa-d712-4111-beda-f7c6ec7910fd-kube-api-access-j2bmx\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001097 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001122 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9wz\" (UniqueName: \"kubernetes.io/projected/84917d34-2512-43fa-86d4-e7c00104371d-kube-api-access-cb9wz\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.001153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001143 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f130cc-b73a-4f66-82b6-0b687aa48686-cert\") pod \"ingress-canary-rrssb\" (UID: \"33f130cc-b73a-4f66-82b6-0b687aa48686\") " pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfdn\" (UniqueName: \"kubernetes.io/projected/f046f2d0-0152-4b1e-856b-55f4eb088650-kube-api-access-pkfdn\") pod \"managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw\" (UID: \"f046f2d0-0152-4b1e-856b-55f4eb088650\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-trusted-ca\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-kube-api-access-rdpt4\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84917d34-2512-43fa-86d4-e7c00104371d-config-volume\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001373 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-tmp\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84917d34-2512-43fa-86d4-e7c00104371d-tmp-dir\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f046f2d0-0152-4b1e-856b-55f4eb088650-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw\" (UID: \"f046f2d0-0152-4b1e-856b-55f4eb088650\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001443 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001465 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtk8t\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-kube-api-access-rtk8t\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.001522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.001516 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.005645 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.005627 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f046f2d0-0152-4b1e-856b-55f4eb088650-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw\" (UID: \"f046f2d0-0152-4b1e-856b-55f4eb088650\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:37.012548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.012516 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfdn\" (UniqueName: \"kubernetes.io/projected/f046f2d0-0152-4b1e-856b-55f4eb088650-kube-api-access-pkfdn\") pod \"managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw\" (UID: \"f046f2d0-0152-4b1e-856b-55f4eb088650\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:37.102239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-certificates\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.102409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-ca\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.102409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-bound-sa-token\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.102409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-hub\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.102564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/984023aa-d712-4111-beda-f7c6ec7910fd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.102564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102482 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.102564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gm5f\" (UniqueName: \"kubernetes.io/projected/33f130cc-b73a-4f66-82b6-0b687aa48686-kube-api-access-5gm5f\") pod \"ingress-canary-rrssb\" (UID: \"33f130cc-b73a-4f66-82b6-0b687aa48686\") " pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.102712 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bmx\" (UniqueName: \"kubernetes.io/projected/984023aa-d712-4111-beda-f7c6ec7910fd-kube-api-access-j2bmx\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.102712 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102611 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.102712 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9wz\" (UniqueName: \"kubernetes.io/projected/84917d34-2512-43fa-86d4-e7c00104371d-kube-api-access-cb9wz\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.102712 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f130cc-b73a-4f66-82b6-0b687aa48686-cert\") pod \"ingress-canary-rrssb\" (UID: \"33f130cc-b73a-4f66-82b6-0b687aa48686\") " pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.102712 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102696 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-trusted-ca\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102728 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-kube-api-access-rdpt4\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84917d34-2512-43fa-86d4-e7c00104371d-config-volume\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-tmp\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84917d34-2512-43fa-86d4-e7c00104371d-tmp-dir\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk8t\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-kube-api-access-rtk8t\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102890 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-certificates\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.102951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.102949 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.103016 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-image-registry-private-configuration\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.103050 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69f1ae7f-7b01-4378-9f18-77df3071c780-ca-trust-extracted\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.103081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-installation-pull-secrets\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.103117 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/984023aa-d712-4111-beda-f7c6ec7910fd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.103160 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.103208 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84917d34-2512-43fa-86d4-e7c00104371d-tmp-dir\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.103394 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.103216 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:31:37.603196921 +0000 UTC m=+34.165361772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:31:37.103748 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.103508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-tmp\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.104628 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.104099 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:31:37.104628 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.104118 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:31:37.104628 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.104170 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:37.604150899 +0000 UTC m=+34.166315760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:31:37.104628 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.104482 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-trusted-ca\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.104910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.104766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69f1ae7f-7b01-4378-9f18-77df3071c780-ca-trust-extracted\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.105532 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.105510 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-ca\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.105679 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.105630 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.106872 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.106852 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-hub\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.107013 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.106937 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-installation-pull-secrets\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.107406 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.107369 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-image-registry-private-configuration\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.108128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.108074 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-klusterlet-config\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.108226 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.108123 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:37.108425 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.108408 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/984023aa-d712-4111-beda-f7c6ec7910fd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.108837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.108808 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f130cc-b73a-4f66-82b6-0b687aa48686-cert\") pod \"ingress-canary-rrssb\" (UID: \"33f130cc-b73a-4f66-82b6-0b687aa48686\") " pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.109015 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.108999 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84917d34-2512-43fa-86d4-e7c00104371d-config-volume\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.113556 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.113513 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk8t\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-kube-api-access-rtk8t\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.113659 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.113593 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-bound-sa-token\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.113864 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.113820 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bmx\" (UniqueName: \"kubernetes.io/projected/984023aa-d712-4111-beda-f7c6ec7910fd-kube-api-access-j2bmx\") pod \"cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl\" (UID: \"984023aa-d712-4111-beda-f7c6ec7910fd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.114060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.114038 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/6a5ce98e-c819-430b-85ef-72f0c7f8efcc-kube-api-access-rdpt4\") pod \"klusterlet-addon-workmgr-6998f87798-hds8z\" (UID: \"6a5ce98e-c819-430b-85ef-72f0c7f8efcc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.115077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.115056 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gm5f\" (UniqueName: \"kubernetes.io/projected/33f130cc-b73a-4f66-82b6-0b687aa48686-kube-api-access-5gm5f\") pod \"ingress-canary-rrssb\" (UID: \"33f130cc-b73a-4f66-82b6-0b687aa48686\") " pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.115989 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.115972 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9wz\" (UniqueName: \"kubernetes.io/projected/84917d34-2512-43fa-86d4-e7c00104371d-kube-api-access-cb9wz\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.149183 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.149153 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" Sep 17 17:31:37.173171 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.173138 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:37.195184 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.195106 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:31:37.236426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.236339 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrssb" Sep 17 17:31:37.281894 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.281865 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:37.455336 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.455312 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw"] Sep 17 17:31:37.460345 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:37.460313 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf046f2d0_0152_4b1e_856b_55f4eb088650.slice/crio-ba6cd69f5613223cc3c4946594a3aed3fe8c24e8049714a282406618a07675bc WatchSource:0}: Error finding container ba6cd69f5613223cc3c4946594a3aed3fe8c24e8049714a282406618a07675bc: Status 404 returned error can't find the container with id ba6cd69f5613223cc3c4946594a3aed3fe8c24e8049714a282406618a07675bc Sep 17 17:31:37.468021 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.467982 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z"] Sep 17 17:31:37.471077 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:37.471044 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5ce98e_c819_430b_85ef_72f0c7f8efcc.slice/crio-6c0098f4615e885b33c1f11fab2fc8094bf4b8bb1e05d86b75e848fe36dfd777 WatchSource:0}: Error finding container 6c0098f4615e885b33c1f11fab2fc8094bf4b8bb1e05d86b75e848fe36dfd777: Status 404 returned error can't find the container with id 6c0098f4615e885b33c1f11fab2fc8094bf4b8bb1e05d86b75e848fe36dfd777 Sep 17 17:31:37.486243 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.486218 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl"] Sep 17 17:31:37.488603 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:37.488582 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod984023aa_d712_4111_beda_f7c6ec7910fd.slice/crio-d7b1e848f184bc7942d10fcda5d88851c82ac10f92de8e5fd6fac2bdbec25219 WatchSource:0}: Error finding container d7b1e848f184bc7942d10fcda5d88851c82ac10f92de8e5fd6fac2bdbec25219: Status 404 returned error can't find the container with id d7b1e848f184bc7942d10fcda5d88851c82ac10f92de8e5fd6fac2bdbec25219 Sep 17 17:31:37.521877 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.521821 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrssb"] Sep 17 17:31:37.524240 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:31:37.524221 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f130cc_b73a_4f66_82b6_0b687aa48686.slice/crio-fb8ccccebb759993b59f012b597f8f6588336cf6471c3b3e662fd53529f3e481 WatchSource:0}: Error finding container fb8ccccebb759993b59f012b597f8f6588336cf6471c3b3e662fd53529f3e481: Status 404 returned error can't find the container with id fb8ccccebb759993b59f012b597f8f6588336cf6471c3b3e662fd53529f3e481 Sep 17 17:31:37.608730 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.608702 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:37.608899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.608866 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:37.608899 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.608880 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:31:37.608899 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.608897 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:31:37.609084 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.608988 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:38.60894614 +0000 UTC m=+35.171111013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:31:37.609084 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.609019 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:31:37.609084 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.609076 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:31:38.609063981 +0000 UTC m=+35.171228835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:31:37.710160 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.710126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:37.710160 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.710157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:37.710383 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.710254 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:37.710383 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.710307 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:32:09.710294734 +0000 UTC m=+66.272459577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:31:37.710383 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.710256 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:37.710554 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.710406 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret podName:d889d683-785d-40d7-a042-9e95c805b529 nodeName:}" failed. No retries permitted until 2026-09-17 17:32:09.71038938 +0000 UTC m=+66.272554221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret") pod "global-pull-secret-syncer-qvfw4" (UID: "d889d683-785d-40d7-a042-9e95c805b529") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:31:37.810965 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:37.810932 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:37.811122 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.811094 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:31:37.811122 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.811119 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:31:37.811243 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.811132 2566 projected.go:196] Error preparing data for projected volume kube-api-access-fq6qv for pod openshift-network-diagnostics/network-check-target-lcqhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:37.811243 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:37.811193 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv podName:eb3e211e-a983-4322-940a-1962fc9705ab nodeName:}" failed. No retries permitted until 2026-09-17 17:32:09.811175432 +0000 UTC m=+66.373340284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6qv" (UniqueName: "kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv") pod "network-check-target-lcqhc" (UID: "eb3e211e-a983-4322-940a-1962fc9705ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:31:38.018081 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.018032 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:38.144444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.144365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:31:38.144590 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.144371 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:31:38.144590 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.144506 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:31:38.147409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.147389 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-js6ht\"" Sep 17 17:31:38.148436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.148308 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 17:31:38.148436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.148324 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 17:31:38.148436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.148350 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 17:31:38.148436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.148352 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 17:31:38.148436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.148411 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gjh79\"" Sep 17 17:31:38.162810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.162789 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:38.293019 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.292980 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrssb" event={"ID":"33f130cc-b73a-4f66-82b6-0b687aa48686","Type":"ContainerStarted","Data":"fb8ccccebb759993b59f012b597f8f6588336cf6471c3b3e662fd53529f3e481"} Sep 17 17:31:38.294289 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.294259 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" event={"ID":"984023aa-d712-4111-beda-f7c6ec7910fd","Type":"ContainerStarted","Data":"d7b1e848f184bc7942d10fcda5d88851c82ac10f92de8e5fd6fac2bdbec25219"} Sep 17 17:31:38.295525 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.295498 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" event={"ID":"f046f2d0-0152-4b1e-856b-55f4eb088650","Type":"ContainerStarted","Data":"ba6cd69f5613223cc3c4946594a3aed3fe8c24e8049714a282406618a07675bc"} Sep 17 17:31:38.296706 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.296684 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" event={"ID":"6a5ce98e-c819-430b-85ef-72f0c7f8efcc","Type":"ContainerStarted","Data":"6c0098f4615e885b33c1f11fab2fc8094bf4b8bb1e05d86b75e848fe36dfd777"} Sep 17 17:31:38.409235 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.409144 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:38.617838 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.617794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:38.618038 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.617862 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:38.618038 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:38.617987 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:31:38.618038 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:38.618000 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:31:38.618038 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:38.617997 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:31:38.618183 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:38.618066 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:40.618049076 +0000 UTC m=+37.180213965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:31:38.618183 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:38.618081 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:31:40.618075026 +0000 UTC m=+37.180239864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:31:38.676714 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:38.676619 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:39.317424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:39.317389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerStarted","Data":"ac1790b04a417f46d44aec69e708ea954cfcc7c0be6d82c04989d143c7f32b49"} Sep 17 17:31:40.332527 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:40.331756 2566 generic.go:356] "Generic (PLEG): container finished" podID="d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe" containerID="ac1790b04a417f46d44aec69e708ea954cfcc7c0be6d82c04989d143c7f32b49" exitCode=0 Sep 17 17:31:40.332527 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:40.331820 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerDied","Data":"ac1790b04a417f46d44aec69e708ea954cfcc7c0be6d82c04989d143c7f32b49"} Sep 17 17:31:40.638218 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:40.638182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:40.638377 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:40.638257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:40.638465 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:40.638401 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:31:40.638465 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:40.638416 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:31:40.638569 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:40.638470 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:44.63845306 +0000 UTC m=+41.200617905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:31:40.638569 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:40.638534 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:31:40.638569 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:40.638565 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:31:44.638554864 +0000 UTC m=+41.200719704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:31:41.339936 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:41.339899 2566 generic.go:356] "Generic (PLEG): container finished" podID="d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe" containerID="7088e8179fb4ef29e36e18c47e4666dc584ca44f9ffb3a6b9bee2a396a9a96f2" exitCode=0 Sep 17 17:31:41.340496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:41.339975 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerDied","Data":"7088e8179fb4ef29e36e18c47e4666dc584ca44f9ffb3a6b9bee2a396a9a96f2"} Sep 17 17:31:41.514629 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:41.514563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:41.640512 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:41.640438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:43.839546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:43.839438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:44.045696 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:44.045663 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:44.674975 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:44.674916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:44.675162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:44.675007 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:44.675162 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:44.675081 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:31:44.675162 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:44.675153 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:31:52.675132764 +0000 UTC m=+49.237297621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:31:44.675162 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:44.675153 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:31:44.675394 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:44.675173 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:31:44.675394 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:44.675208 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:31:52.675197584 +0000 UTC m=+49.237362441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:31:48.363355 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.363313 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" event={"ID":"f046f2d0-0152-4b1e-856b-55f4eb088650","Type":"ContainerStarted","Data":"e3584ee0cd0b51f74a957b2bbc9cf3209a3694fdadd2dd872491b35654d6c1b1"} Sep 17 17:31:48.364825 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.364802 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" event={"ID":"6a5ce98e-c819-430b-85ef-72f0c7f8efcc","Type":"ContainerStarted","Data":"40a2d91626a3af91aeb2be261cf09f9da15f9d7c17dde3a27e07c1946a2876cf"} Sep 17 17:31:48.365014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.364995 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:48.366756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.366737 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:31:48.368465 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.368447 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q85q4" event={"ID":"d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe","Type":"ContainerStarted","Data":"7ca745834ffcd2ad81a2bb4811bb912d0dde4ccf7994582e820a96563e7fabed"} Sep 17 17:31:48.369910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.369888 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrssb" event={"ID":"33f130cc-b73a-4f66-82b6-0b687aa48686","Type":"ContainerStarted","Data":"f85c2375e571784599527fc3b290c1a57551521aa603c207fb470bbf0d7a195f"} Sep 17 17:31:48.371781 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.371754 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" event={"ID":"984023aa-d712-4111-beda-f7c6ec7910fd","Type":"ContainerStarted","Data":"2835bf42f862fa47b8f949b57b210eaca9ee1492de6b383a265cdb607e1ad94e"} Sep 17 17:31:48.371781 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.371783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" event={"ID":"984023aa-d712-4111-beda-f7c6ec7910fd","Type":"ContainerStarted","Data":"6f40c0b6c0d84a6575c9619e93282393a2e282b19898e7a34197fe6e70699715"} Sep 17 17:31:48.371903 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.371799 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" event={"ID":"984023aa-d712-4111-beda-f7c6ec7910fd","Type":"ContainerStarted","Data":"1069d46e0fe3c7ddbc30bd6552827feae54fde82a340dbe33051e7f9398bd901"} Sep 17 17:31:48.382043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.382004 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" podStartSLOduration=15.560795689999999 podStartE2EDuration="25.381991356s" totalImagesPullingTime="9.821195666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:48.381000428 +0000 UTC m=+44.943165282" watchObservedRunningTime="2026-09-17 17:31:48.381991356 +0000 UTC m=+44.944156210" Sep 17 17:31:48.405392 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.405347 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" podStartSLOduration=15.537226764 podStartE2EDuration="25.405333848s" totalImagesPullingTime="9.868107084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:48.404924633 +0000 UTC m=+44.967089495" watchObservedRunningTime="2026-09-17 17:31:48.405333848 +0000 UTC m=+44.967498709" Sep 17 17:31:48.429602 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.429555 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" podStartSLOduration=15.60365235 podStartE2EDuration="25.429537852s" totalImagesPullingTime="9.825885502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:48.429443625 +0000 UTC m=+44.991608485" watchObservedRunningTime="2026-09-17 17:31:48.429537852 +0000 UTC m=+44.991702713" Sep 17 17:31:48.447430 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.447389 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rrssb" podStartSLOduration=2.657922384 podStartE2EDuration="12.447375022s" totalImagesPullingTime="9.789452638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:48.44623193 +0000 UTC m=+45.008396790" watchObservedRunningTime="2026-09-17 17:31:48.447375022 +0000 UTC m=+45.009539883" Sep 17 17:31:48.482786 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.482285 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q85q4" podStartSLOduration=14.260166616 podStartE2EDuration="44.482268264s" totalImagesPullingTime="30.222101648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:48.479146501 +0000 UTC m=+45.041311362" watchObservedRunningTime="2026-09-17 17:31:48.482268264 +0000 UTC m=+45.044433125" Sep 17 17:31:48.757034 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.757002 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:48.906321 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:48.906288 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:50.065486 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:50.065456 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:50.182150 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:50.182115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:51.092139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:51.092105 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:51.223963 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:51.223935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:52.734569 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:52.734535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:31:52.734982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:52.734582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:31:52.734982 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:52.734699 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:31:52.734982 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:52.734789 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:32:08.734768523 +0000 UTC m=+65.296933362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:31:52.734982 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:52.734804 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:31:52.734982 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:52.734819 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:31:52.734982 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:31:52.734870 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:32:08.734858325 +0000 UTC m=+65.297023167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:31:54.379872 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:54.379817 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:31:54.483951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:31:54.483919 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" limit=1024 Sep 17 17:32:01.282972 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:01.282937 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5zwsd" Sep 17 17:32:08.748616 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:08.748577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:32:08.749088 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:08.748675 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:32:08.749088 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:08.748728 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:32:08.749088 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:08.748749 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:32:08.749088 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:08.748789 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:32:08.749088 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:08.748805 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:32:40.748788892 +0000 UTC m=+97.310953729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:32:08.749088 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:08.748837 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:32:40.748822654 +0000 UTC m=+97.310987511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:32:09.755686 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.755647 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:32:09.755686 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.755685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:32:09.759142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.759122 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 17:32:09.759751 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.759735 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 17:32:09.766338 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:09.766317 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 17:32:09.766416 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:09.766385 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:13.766363572 +0000 UTC m=+130.328528432 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : secret "metrics-daemon-secret" not found Sep 17 17:32:09.769138 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.769121 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d889d683-785d-40d7-a042-9e95c805b529-original-pull-secret\") pod \"global-pull-secret-syncer-qvfw4\" (UID: \"d889d683-785d-40d7-a042-9e95c805b529\") " pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:32:09.856231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.856193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:32:09.861925 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.861899 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 17:32:09.868837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.868817 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 17:32:09.879222 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.879195 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6qv\" (UniqueName: \"kubernetes.io/projected/eb3e211e-a983-4322-940a-1962fc9705ab-kube-api-access-fq6qv\") pod \"network-check-target-lcqhc\" (UID: \"eb3e211e-a983-4322-940a-1962fc9705ab\") " pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:32:09.964559 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.964523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qvfw4" Sep 17 17:32:09.972587 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.972561 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gjh79\"" Sep 17 17:32:09.980108 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:09.980080 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:32:10.285374 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:10.285303 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qvfw4"] Sep 17 17:32:10.288492 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:32:10.288458 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd889d683_785d_40d7_a042_9e95c805b529.slice/crio-354f9289a43b233ec1bd010f4c1e8f03e91a8cb3ae408fb5addfed0ca34fabf4 WatchSource:0}: Error finding container 354f9289a43b233ec1bd010f4c1e8f03e91a8cb3ae408fb5addfed0ca34fabf4: Status 404 returned error can't find the container with id 354f9289a43b233ec1bd010f4c1e8f03e91a8cb3ae408fb5addfed0ca34fabf4 Sep 17 17:32:10.308280 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:10.308257 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lcqhc"] Sep 17 17:32:10.311028 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:32:10.311004 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3e211e_a983_4322_940a_1962fc9705ab.slice/crio-c7735909291dfab615cccb2ad8a39a9560115fd34a0428858ac0f358d18b059c WatchSource:0}: Error finding container c7735909291dfab615cccb2ad8a39a9560115fd34a0428858ac0f358d18b059c: Status 404 returned error can't find the container with id c7735909291dfab615cccb2ad8a39a9560115fd34a0428858ac0f358d18b059c Sep 17 17:32:10.439892 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:10.439860 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qvfw4" event={"ID":"d889d683-785d-40d7-a042-9e95c805b529","Type":"ContainerStarted","Data":"354f9289a43b233ec1bd010f4c1e8f03e91a8cb3ae408fb5addfed0ca34fabf4"} Sep 17 17:32:10.440991 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:10.440946 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lcqhc" event={"ID":"eb3e211e-a983-4322-940a-1962fc9705ab","Type":"ContainerStarted","Data":"c7735909291dfab615cccb2ad8a39a9560115fd34a0428858ac0f358d18b059c"} Sep 17 17:32:15.458800 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:15.458760 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qvfw4" event={"ID":"d889d683-785d-40d7-a042-9e95c805b529","Type":"ContainerStarted","Data":"765e7bbb6c621cf74c8c23188f29269658526b9b218dc69591942ea78d0b9400"} Sep 17 17:32:15.460339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:15.460309 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lcqhc" event={"ID":"eb3e211e-a983-4322-940a-1962fc9705ab","Type":"ContainerStarted","Data":"dc1a9c1c12068b82accb4e0eb4f0a61c138ff87d5941cd396883384bfb104182"} Sep 17 17:32:15.460454 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:15.460369 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:32:15.477312 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:15.477271 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qvfw4" podStartSLOduration=66.90280304 podStartE2EDuration="1m11.47725728s" totalImagesPullingTime="4.57445424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:32:15.476472492 +0000 UTC m=+72.038637352" watchObservedRunningTime="2026-09-17 17:32:15.47725728 +0000 UTC m=+72.039422140" Sep 17 17:32:15.499240 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:15.499197 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lcqhc" podStartSLOduration=66.950233596 podStartE2EDuration="1m11.49918529s" totalImagesPullingTime="4.548951694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:32:15.498977433 +0000 UTC m=+72.061142292" watchObservedRunningTime="2026-09-17 17:32:15.49918529 +0000 UTC m=+72.061350154" Sep 17 17:32:40.779969 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:40.779927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:32:40.780366 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:40.779996 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:32:40.780366 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:40.780072 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 17:32:40.780366 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:40.780146 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls podName:84917d34-2512-43fa-86d4-e7c00104371d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:44.780130491 +0000 UTC m=+161.342295328 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls") pod "dns-default-wm2jb" (UID: "84917d34-2512-43fa-86d4-e7c00104371d") : secret "dns-default-metrics-tls" not found Sep 17 17:32:40.780366 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:40.780073 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 17:32:40.780366 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:40.780172 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8669fdd6cf-jcw8m: secret "image-registry-tls" not found Sep 17 17:32:40.780366 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:32:40.780227 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls podName:69f1ae7f-7b01-4378-9f18-77df3071c780 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:44.780214272 +0000 UTC m=+161.342379113 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls") pod "image-registry-8669fdd6cf-jcw8m" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780") : secret "image-registry-tls" not found Sep 17 17:32:46.464993 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:32:46.464966 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lcqhc" Sep 17 17:33:10.112597 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.112562 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g8qdz"] Sep 17 17:33:10.115610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.115594 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.119653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.118441 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 17:33:10.119653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.118672 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 17:33:10.119826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.119750 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 17:33:10.119870 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.119854 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tztbm\"" Sep 17 17:33:10.122520 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.122505 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 17:33:10.140066 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.140031 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g8qdz"] Sep 17 17:33:10.297012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.296978 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/162124ac-91f3-43bd-b7b1-018b7616814f-crio-socket\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.297012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.297017 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9772\" (UniqueName: \"kubernetes.io/projected/162124ac-91f3-43bd-b7b1-018b7616814f-kube-api-access-x9772\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.297239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.297037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/162124ac-91f3-43bd-b7b1-018b7616814f-data-volume\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.297239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.297124 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.297239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.297150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398041 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.397977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398041 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.398015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.398048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/162124ac-91f3-43bd-b7b1-018b7616814f-crio-socket\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.398072 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9772\" (UniqueName: \"kubernetes.io/projected/162124ac-91f3-43bd-b7b1-018b7616814f-kube-api-access-x9772\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.398088 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/162124ac-91f3-43bd-b7b1-018b7616814f-data-volume\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398162 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.398090 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 17:33:10.398281 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.398169 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/162124ac-91f3-43bd-b7b1-018b7616814f-crio-socket\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.398281 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.398183 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:10.898162218 +0000 UTC m=+127.460327058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : configmap "kube-rbac-proxy" not found Sep 17 17:33:10.398281 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.398215 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 17:33:10.398281 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.398276 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:10.898263443 +0000 UTC m=+127.460428280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : secret "insights-runtime-extractor-tls" not found Sep 17 17:33:10.398408 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.398389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/162124ac-91f3-43bd-b7b1-018b7616814f-data-volume\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.409670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.409638 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9772\" (UniqueName: \"kubernetes.io/projected/162124ac-91f3-43bd-b7b1-018b7616814f-kube-api-access-x9772\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.903216 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.903180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.903424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:10.903225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:10.903424 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.903308 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 17:33:10.903424 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.903383 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:11.903366664 +0000 UTC m=+128.465531502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : configmap "kube-rbac-proxy" not found Sep 17 17:33:10.903424 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.903407 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 17:33:10.903613 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:10.903469 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:11.903453482 +0000 UTC m=+128.465618320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : secret "insights-runtime-extractor-tls" not found Sep 17 17:33:11.909416 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:11.909383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:11.909416 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:11.909418 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:11.909811 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:11.909504 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 17:33:11.909811 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:11.909522 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 17:33:11.909811 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:11.909569 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:13.909556668 +0000 UTC m=+130.471721506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : secret "insights-runtime-extractor-tls" not found Sep 17 17:33:11.909811 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:11.909584 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:13.909577204 +0000 UTC m=+130.471742042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : configmap "kube-rbac-proxy" not found Sep 17 17:33:13.036006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:13.035979 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vppd9_eb353df6-c110-4406-9efb-e29e653be6dc/dns-node-resolver/0.log" Sep 17 17:33:13.637089 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:13.637061 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7g2lv_2b69f4eb-f959-4d3c-9331-3a8ddf0af363/node-ca/0.log" Sep 17 17:33:13.824208 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:13.824177 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:33:13.824351 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:13.824288 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 17:33:13.824351 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:13.824340 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs podName:fc9a2b4e-2ae6-496a-9db8-abfbed21ab09 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:15.824326805 +0000 UTC m=+252.386491642 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs") pod "network-metrics-daemon-lbj98" (UID: "fc9a2b4e-2ae6-496a-9db8-abfbed21ab09") : secret "metrics-daemon-secret" not found Sep 17 17:33:13.925430 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:13.925349 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:13.925430 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:13.925394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:13.925622 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:13.925473 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 17:33:13.925622 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:13.925531 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:17.925516462 +0000 UTC m=+134.487681305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : configmap "kube-rbac-proxy" not found Sep 17 17:33:13.925622 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:13.925543 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 17:33:13.925622 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:13.925595 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:17.92557889 +0000 UTC m=+134.487743748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : secret "insights-runtime-extractor-tls" not found Sep 17 17:33:14.836065 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:14.836040 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrssb_33f130cc-b73a-4f66-82b6-0b687aa48686/serve-healthcheck-canary/0.log" Sep 17 17:33:17.952542 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:17.952495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:17.952542 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:17.952551 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:17.953067 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:17.952589 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 17:33:17.953067 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:17.952665 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 17:33:17.953067 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:17.952668 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:25.95265287 +0000 UTC m=+142.514817708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : configmap "kube-rbac-proxy" not found Sep 17 17:33:17.953067 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:17.952712 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:25.952699869 +0000 UTC m=+142.514864707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : secret "insights-runtime-extractor-tls" not found Sep 17 17:33:26.012783 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:26.012752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:26.012783 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:26.012788 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:26.013255 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:26.012897 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 17:33:26.013255 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:26.012980 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls podName:162124ac-91f3-43bd-b7b1-018b7616814f nodeName:}" failed. No retries permitted until 2026-09-17 17:33:42.012943582 +0000 UTC m=+158.575108420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8qdz" (UID: "162124ac-91f3-43bd-b7b1-018b7616814f") : secret "insights-runtime-extractor-tls" not found Sep 17 17:33:26.013328 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:26.013258 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/162124ac-91f3-43bd-b7b1-018b7616814f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:39.859351 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:39.859305 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" Sep 17 17:33:39.929072 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:39.929034 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wm2jb" podUID="84917d34-2512-43fa-86d4-e7c00104371d" Sep 17 17:33:40.705988 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:40.705464 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wm2jb" Sep 17 17:33:40.705988 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:40.705903 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:33:41.157092 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:41.157050 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lbj98" podUID="fc9a2b4e-2ae6-496a-9db8-abfbed21ab09" Sep 17 17:33:42.029627 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:42.029585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:42.031962 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:42.031929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/162124ac-91f3-43bd-b7b1-018b7616814f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8qdz\" (UID: \"162124ac-91f3-43bd-b7b1-018b7616814f\") " pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:42.226972 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:42.226926 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g8qdz" Sep 17 17:33:42.501693 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:42.501551 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g8qdz"] Sep 17 17:33:42.504117 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:33:42.504091 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162124ac_91f3_43bd_b7b1_018b7616814f.slice/crio-7df05f9d35f95d70fb7b01849885d0df772d8d9802574ec6a2736145101d3709 WatchSource:0}: Error finding container 7df05f9d35f95d70fb7b01849885d0df772d8d9802574ec6a2736145101d3709: Status 404 returned error can't find the container with id 7df05f9d35f95d70fb7b01849885d0df772d8d9802574ec6a2736145101d3709 Sep 17 17:33:42.712097 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:42.712063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8qdz" event={"ID":"162124ac-91f3-43bd-b7b1-018b7616814f","Type":"ContainerStarted","Data":"4a9625f368b7c976eea81b8a1ea5d4494c6f3b9e72f9400d60391b60db495674"} Sep 17 17:33:42.712269 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:42.712120 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8qdz" event={"ID":"162124ac-91f3-43bd-b7b1-018b7616814f","Type":"ContainerStarted","Data":"7df05f9d35f95d70fb7b01849885d0df772d8d9802574ec6a2736145101d3709"} Sep 17 17:33:43.717375 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:43.717336 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8qdz" event={"ID":"162124ac-91f3-43bd-b7b1-018b7616814f","Type":"ContainerStarted","Data":"f8e17ceab7b9fed4040ea11775e3cb928ac8f12e1bd400358872d79d85c2ddae"} Sep 17 17:33:44.722117 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.722089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8qdz" event={"ID":"162124ac-91f3-43bd-b7b1-018b7616814f","Type":"ContainerStarted","Data":"d550b3eae2d5cb59eb6241423eaabcd2187733025d6a4130a66afafc4fed10e3"} Sep 17 17:33:44.740354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.740315 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g8qdz" podStartSLOduration=32.710387226 podStartE2EDuration="34.740300246s" totalImagesPullingTime="2.02991302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:44.739833536 +0000 UTC m=+161.301998389" watchObservedRunningTime="2026-09-17 17:33:44.740300246 +0000 UTC m=+161.302465106" Sep 17 17:33:44.851458 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.851423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:33:44.851622 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.851472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:33:44.855989 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.855236 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"image-registry-8669fdd6cf-jcw8m\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:33:44.857871 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.857848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84917d34-2512-43fa-86d4-e7c00104371d-metrics-tls\") pod \"dns-default-wm2jb\" (UID: \"84917d34-2512-43fa-86d4-e7c00104371d\") " pod="openshift-dns/dns-default-wm2jb" Sep 17 17:33:44.910282 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.910209 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fv9xq\"" Sep 17 17:33:44.910404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.910300 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b7bq9\"" Sep 17 17:33:44.917373 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.917354 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:33:44.917463 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:44.917448 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wm2jb" Sep 17 17:33:45.197466 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.197375 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wm2jb"] Sep 17 17:33:45.200665 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:33:45.200642 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84917d34_2512_43fa_86d4_e7c00104371d.slice/crio-40239d95a75e9a349ccfc6f20ac90b82931a12f60af0d1acca68365ce6df4356 WatchSource:0}: Error finding container 40239d95a75e9a349ccfc6f20ac90b82931a12f60af0d1acca68365ce6df4356: Status 404 returned error can't find the container with id 40239d95a75e9a349ccfc6f20ac90b82931a12f60af0d1acca68365ce6df4356 Sep 17 17:33:45.207478 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.207454 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8669fdd6cf-jcw8m"] Sep 17 17:33:45.209836 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:33:45.209816 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f1ae7f_7b01_4378_9f18_77df3071c780.slice/crio-bdb16a8858e24e1bc7b75ee4379fcca827028378a3b9c5f7d72e7269840f092a WatchSource:0}: Error finding container bdb16a8858e24e1bc7b75ee4379fcca827028378a3b9c5f7d72e7269840f092a: Status 404 returned error can't find the container with id bdb16a8858e24e1bc7b75ee4379fcca827028378a3b9c5f7d72e7269840f092a Sep 17 17:33:45.727452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.727155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" event={"ID":"69f1ae7f-7b01-4378-9f18-77df3071c780","Type":"ContainerStarted","Data":"6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296"} Sep 17 17:33:45.727452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.727212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" event={"ID":"69f1ae7f-7b01-4378-9f18-77df3071c780","Type":"ContainerStarted","Data":"bdb16a8858e24e1bc7b75ee4379fcca827028378a3b9c5f7d72e7269840f092a"} Sep 17 17:33:45.727452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.727268 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:33:45.728588 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.728562 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wm2jb" event={"ID":"84917d34-2512-43fa-86d4-e7c00104371d","Type":"ContainerStarted","Data":"40239d95a75e9a349ccfc6f20ac90b82931a12f60af0d1acca68365ce6df4356"} Sep 17 17:33:45.753057 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:45.753005 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" podStartSLOduration=161.752985711 podStartE2EDuration="2m41.752985711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:45.751271698 +0000 UTC m=+162.313436559" watchObservedRunningTime="2026-09-17 17:33:45.752985711 +0000 UTC m=+162.315150568" Sep 17 17:33:47.736231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.736198 2566 generic.go:356] "Generic (PLEG): container finished" podID="6a5ce98e-c819-430b-85ef-72f0c7f8efcc" containerID="40a2d91626a3af91aeb2be261cf09f9da15f9d7c17dde3a27e07c1946a2876cf" exitCode=1 Sep 17 17:33:47.736596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.736276 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" event={"ID":"6a5ce98e-c819-430b-85ef-72f0c7f8efcc","Type":"ContainerDied","Data":"40a2d91626a3af91aeb2be261cf09f9da15f9d7c17dde3a27e07c1946a2876cf"} Sep 17 17:33:47.736695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.736672 2566 scope.go:117] "RemoveContainer" containerID="40a2d91626a3af91aeb2be261cf09f9da15f9d7c17dde3a27e07c1946a2876cf" Sep 17 17:33:47.738036 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.738017 2566 generic.go:356] "Generic (PLEG): container finished" podID="f046f2d0-0152-4b1e-856b-55f4eb088650" containerID="e3584ee0cd0b51f74a957b2bbc9cf3209a3694fdadd2dd872491b35654d6c1b1" exitCode=255 Sep 17 17:33:47.738118 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.738087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" event={"ID":"f046f2d0-0152-4b1e-856b-55f4eb088650","Type":"ContainerDied","Data":"e3584ee0cd0b51f74a957b2bbc9cf3209a3694fdadd2dd872491b35654d6c1b1"} Sep 17 17:33:47.738400 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.738382 2566 scope.go:117] "RemoveContainer" containerID="e3584ee0cd0b51f74a957b2bbc9cf3209a3694fdadd2dd872491b35654d6c1b1" Sep 17 17:33:47.740043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.740019 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wm2jb" event={"ID":"84917d34-2512-43fa-86d4-e7c00104371d","Type":"ContainerStarted","Data":"821410be3d9fb41316d748b9e39fc3e3de00e2424396dbbbd6655351a0fe8779"} Sep 17 17:33:47.740153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.740065 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wm2jb" event={"ID":"84917d34-2512-43fa-86d4-e7c00104371d","Type":"ContainerStarted","Data":"79a4cb849b2f73a3246bdf70131a3cad96a657726da5bf503dbe94ec4aeb26b4"} Sep 17 17:33:47.740203 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.740152 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wm2jb" Sep 17 17:33:47.814181 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:47.814134 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wm2jb" podStartSLOduration=130.273406322 podStartE2EDuration="2m11.814118785s" totalImagesPullingTime="1.540712463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:47.789712663 +0000 UTC m=+164.351877526" watchObservedRunningTime="2026-09-17 17:33:47.814118785 +0000 UTC m=+164.376283639" Sep 17 17:33:48.365883 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:48.365837 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:33:48.745000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:48.744898 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" event={"ID":"6a5ce98e-c819-430b-85ef-72f0c7f8efcc","Type":"ContainerStarted","Data":"0211eb1e09a79d866daeb7e1faabe98e57edbaa51590fae2bdc22e708fefb85f"} Sep 17 17:33:48.745423 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:48.745084 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:33:48.745713 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:48.745687 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6998f87798-hds8z" Sep 17 17:33:48.746752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:48.746730 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fdbc668dd-kpbzw" event={"ID":"f046f2d0-0152-4b1e-856b-55f4eb088650","Type":"ContainerStarted","Data":"1c9b6d991bcda3163ba7cb67d12018de64799b8cdf125595ad120616d003ebd0"} Sep 17 17:33:56.143924 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.143890 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:33:56.249122 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.249091 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7hs8h"] Sep 17 17:33:56.252125 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.252110 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.261330 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.261286 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 17:33:56.261445 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.261399 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 17:33:56.262223 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.262207 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 17:33:56.262311 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.262212 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xkwcs\"" Sep 17 17:33:56.262492 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.262477 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 17:33:56.262580 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.262566 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 17:33:56.262637 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.262625 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 17:33:56.331046 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331026 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc72b41-3410-4e68-8c43-c7f487adde2d-metrics-client-ca\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331059 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-textfile\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331096 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331232 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331140 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-sys\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331232 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331163 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331232 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331186 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331232 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrgr\" (UniqueName: \"kubernetes.io/projected/6cc72b41-3410-4e68-8c43-c7f487adde2d-kube-api-access-bmrgr\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331232 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331225 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-root\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.331449 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.331346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-wtmp\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.431869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-root\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.431869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-wtmp\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.431869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc72b41-3410-4e68-8c43-c7f487adde2d-metrics-client-ca\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-root\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-textfile\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.431997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-sys\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432044 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-wtmp\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432083 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrgr\" (UniqueName: \"kubernetes.io/projected/6cc72b41-3410-4e68-8c43-c7f487adde2d-kube-api-access-bmrgr\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432090 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cc72b41-3410-4e68-8c43-c7f487adde2d-sys\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432546 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:56.432150 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 17 17:33:56.432546 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:56.432225 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls podName:6cc72b41-3410-4e68-8c43-c7f487adde2d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:56.932203946 +0000 UTC m=+173.494368798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls") pod "node-exporter-7hs8h" (UID: "6cc72b41-3410-4e68-8c43-c7f487adde2d") : secret "node-exporter-tls" not found Sep 17 17:33:56.432546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-textfile\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432535 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.432700 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.432560 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc72b41-3410-4e68-8c43-c7f487adde2d-metrics-client-ca\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.434279 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.434260 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.443446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.443428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrgr\" (UniqueName: \"kubernetes.io/projected/6cc72b41-3410-4e68-8c43-c7f487adde2d-kube-api-access-bmrgr\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.934575 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:56.934542 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:56.934744 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:56.934684 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 17 17:33:56.934790 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:33:56.934755 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls podName:6cc72b41-3410-4e68-8c43-c7f487adde2d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:57.934741036 +0000 UTC m=+174.496905874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls") pod "node-exporter-7hs8h" (UID: "6cc72b41-3410-4e68-8c43-c7f487adde2d") : secret "node-exporter-tls" not found Sep 17 17:33:57.748592 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:57.748567 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wm2jb" Sep 17 17:33:57.943207 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:57.943176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:57.945350 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:57.945333 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc72b41-3410-4e68-8c43-c7f487adde2d-node-exporter-tls\") pod \"node-exporter-7hs8h\" (UID: \"6cc72b41-3410-4e68-8c43-c7f487adde2d\") " pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:58.060343 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:58.060319 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hs8h" Sep 17 17:33:58.070902 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:33:58.070876 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc72b41_3410_4e68_8c43_c7f487adde2d.slice/crio-90c9ac78f6fa17c2c320ab465b99b4b02345326d5e94c30b69869b8147f318c0 WatchSource:0}: Error finding container 90c9ac78f6fa17c2c320ab465b99b4b02345326d5e94c30b69869b8147f318c0: Status 404 returned error can't find the container with id 90c9ac78f6fa17c2c320ab465b99b4b02345326d5e94c30b69869b8147f318c0 Sep 17 17:33:58.779694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:58.779650 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hs8h" event={"ID":"6cc72b41-3410-4e68-8c43-c7f487adde2d","Type":"ContainerStarted","Data":"90c9ac78f6fa17c2c320ab465b99b4b02345326d5e94c30b69869b8147f318c0"} Sep 17 17:33:59.784056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:59.784019 2566 generic.go:356] "Generic (PLEG): container finished" podID="6cc72b41-3410-4e68-8c43-c7f487adde2d" containerID="78d49398fc95a205f458b6386e86d8d2ac3d2d883757e1bb0dc46a001eede8ff" exitCode=0 Sep 17 17:33:59.784422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:33:59.784067 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hs8h" event={"ID":"6cc72b41-3410-4e68-8c43-c7f487adde2d","Type":"ContainerDied","Data":"78d49398fc95a205f458b6386e86d8d2ac3d2d883757e1bb0dc46a001eede8ff"} Sep 17 17:34:00.789518 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:00.789478 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hs8h" event={"ID":"6cc72b41-3410-4e68-8c43-c7f487adde2d","Type":"ContainerStarted","Data":"09ff73a057cbcac3fd822147c077e76d1b9c2eae8c005dee39c185141f76c1f8"} Sep 17 17:34:00.789907 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:00.789541 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hs8h" event={"ID":"6cc72b41-3410-4e68-8c43-c7f487adde2d","Type":"ContainerStarted","Data":"6e8d4e5a0734d0f743073553f05df30b3cae35c52edc13f6e524716183b6529f"} Sep 17 17:34:00.816571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:00.816513 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7hs8h" podStartSLOduration=3.912913468 podStartE2EDuration="4.816498253s" totalImagesPullingTime="820.193995ms" totalInitContainerRuntime="83.39079ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:00.814699932 +0000 UTC m=+177.376864819" watchObservedRunningTime="2026-09-17 17:34:00.816498253 +0000 UTC m=+177.378663112" Sep 17 17:34:04.921689 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:04.921649 2566 patch_prober.go:28] interesting pod/image-registry-8669fdd6cf-jcw8m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 17:34:04.922067 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:04.921709 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:34:06.736275 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:06.736250 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:34:06.899772 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:06.899733 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8669fdd6cf-jcw8m"] Sep 17 17:34:27.196854 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:27.196817 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" podUID="984023aa-d712-4111-beda-f7c6ec7910fd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 17 17:34:31.920703 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:31.920647 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" containerName="registry" containerID="cri-o://6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296" gracePeriod=30 Sep 17 17:34:32.154591 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.154571 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:34:32.288196 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288124 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-installation-pull-secrets\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288196 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288166 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69f1ae7f-7b01-4378-9f18-77df3071c780-ca-trust-extracted\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288355 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtk8t\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-kube-api-access-rtk8t\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288401 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288374 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-certificates\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288440 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288412 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288495 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288441 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-bound-sa-token\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288495 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288490 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-trusted-ca\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288591 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288526 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-image-registry-private-configuration\") pod \"69f1ae7f-7b01-4378-9f18-77df3071c780\" (UID: \"69f1ae7f-7b01-4378-9f18-77df3071c780\") " Sep 17 17:34:32.288869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.288796 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:34:32.289287 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.289231 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:34:32.290926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.290890 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:34:32.291157 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.291135 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-kube-api-access-rtk8t" (OuterVolumeSpecName: "kube-api-access-rtk8t") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "kube-api-access-rtk8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:34:32.291237 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.291180 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:34:32.291237 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.291201 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:34:32.291321 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.291251 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:34:32.297503 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.297479 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f1ae7f-7b01-4378-9f18-77df3071c780-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "69f1ae7f-7b01-4378-9f18-77df3071c780" (UID: "69f1ae7f-7b01-4378-9f18-77df3071c780"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:34:32.389304 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389282 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-trusted-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389304 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389301 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-image-registry-private-configuration\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389311 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69f1ae7f-7b01-4378-9f18-77df3071c780-installation-pull-secrets\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389321 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69f1ae7f-7b01-4378-9f18-77df3071c780-ca-trust-extracted\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389329 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtk8t\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-kube-api-access-rtk8t\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389337 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-certificates\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389345 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-registry-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.389413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.389352 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69f1ae7f-7b01-4378-9f18-77df3071c780-bound-sa-token\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:34:32.890310 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.890276 2566 generic.go:356] "Generic (PLEG): container finished" podID="69f1ae7f-7b01-4378-9f18-77df3071c780" containerID="6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296" exitCode=0 Sep 17 17:34:32.890499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.890317 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" event={"ID":"69f1ae7f-7b01-4378-9f18-77df3071c780","Type":"ContainerDied","Data":"6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296"} Sep 17 17:34:32.890499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.890325 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" Sep 17 17:34:32.890499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.890358 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8669fdd6cf-jcw8m" event={"ID":"69f1ae7f-7b01-4378-9f18-77df3071c780","Type":"ContainerDied","Data":"bdb16a8858e24e1bc7b75ee4379fcca827028378a3b9c5f7d72e7269840f092a"} Sep 17 17:34:32.890499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.890379 2566 scope.go:117] "RemoveContainer" containerID="6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296" Sep 17 17:34:32.898708 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.898692 2566 scope.go:117] "RemoveContainer" containerID="6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296" Sep 17 17:34:32.898995 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:34:32.898976 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296\": container with ID starting with 6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296 not found: ID does not exist" containerID="6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296" Sep 17 17:34:32.899072 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.899009 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296"} err="failed to get container status \"6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296\": rpc error: code = NotFound desc = could not find container \"6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296\": container with ID starting with 6f9fe40b6416966d554d1cee31e7b323e2a1fe873b0ccdb854f1dc4f9d04e296 not found: ID does not exist" Sep 17 17:34:32.912994 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.912969 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8669fdd6cf-jcw8m"] Sep 17 17:34:32.918577 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:32.918558 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8669fdd6cf-jcw8m"] Sep 17 17:34:34.148130 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:34.148099 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" path="/var/lib/kubelet/pods/69f1ae7f-7b01-4378-9f18-77df3071c780/volumes" Sep 17 17:34:37.196459 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:37.196419 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" podUID="984023aa-d712-4111-beda-f7c6ec7910fd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 17 17:34:47.197044 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.197001 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" podUID="984023aa-d712-4111-beda-f7c6ec7910fd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 17 17:34:47.197453 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.197075 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" Sep 17 17:34:47.197523 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.197506 2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2835bf42f862fa47b8f949b57b210eaca9ee1492de6b383a265cdb607e1ad94e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 17 17:34:47.197566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.197542 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" podUID="984023aa-d712-4111-beda-f7c6ec7910fd" containerName="service-proxy" containerID="cri-o://2835bf42f862fa47b8f949b57b210eaca9ee1492de6b383a265cdb607e1ad94e" gracePeriod=30 Sep 17 17:34:47.943016 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.942983 2566 generic.go:356] "Generic (PLEG): container finished" podID="984023aa-d712-4111-beda-f7c6ec7910fd" containerID="2835bf42f862fa47b8f949b57b210eaca9ee1492de6b383a265cdb607e1ad94e" exitCode=2 Sep 17 17:34:47.943173 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.943025 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" event={"ID":"984023aa-d712-4111-beda-f7c6ec7910fd","Type":"ContainerDied","Data":"2835bf42f862fa47b8f949b57b210eaca9ee1492de6b383a265cdb607e1ad94e"} Sep 17 17:34:47.943173 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:34:47.943057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4cd6c96-gzzkl" event={"ID":"984023aa-d712-4111-beda-f7c6ec7910fd","Type":"ContainerStarted","Data":"cc91dbc31e832c9af794ca729e656cf2abb833590541f92806bee2140256d27b"} Sep 17 17:35:15.884156 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:15.884118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:35:15.886329 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:15.886312 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a2b4e-2ae6-496a-9db8-abfbed21ab09-metrics-certs\") pod \"network-metrics-daemon-lbj98\" (UID: \"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09\") " pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:35:15.947286 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:15.947250 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-js6ht\"" Sep 17 17:35:15.954444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:15.954424 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lbj98" Sep 17 17:35:16.225052 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:16.224974 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lbj98"] Sep 17 17:35:16.227859 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:35:16.227831 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9a2b4e_2ae6_496a_9db8_abfbed21ab09.slice/crio-8d9c1e1c8aab83dca54a729cc942973c67bcaa983c7fe8cd9f35fc57c9a965b2 WatchSource:0}: Error finding container 8d9c1e1c8aab83dca54a729cc942973c67bcaa983c7fe8cd9f35fc57c9a965b2: Status 404 returned error can't find the container with id 8d9c1e1c8aab83dca54a729cc942973c67bcaa983c7fe8cd9f35fc57c9a965b2 Sep 17 17:35:17.034303 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:17.034272 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lbj98" event={"ID":"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09","Type":"ContainerStarted","Data":"8d9c1e1c8aab83dca54a729cc942973c67bcaa983c7fe8cd9f35fc57c9a965b2"} Sep 17 17:35:18.039601 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:18.039569 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lbj98" event={"ID":"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09","Type":"ContainerStarted","Data":"a548a82dffa639a98abee98bbdadaa9b86ddd1f7e1bec8eafe651b4a2c9c8e10"} Sep 17 17:35:18.039983 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:18.039625 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lbj98" event={"ID":"fc9a2b4e-2ae6-496a-9db8-abfbed21ab09","Type":"ContainerStarted","Data":"023f9546b3484ff1c2af87d6a96e0fc51f598c53b38440ea009de55df19cdf9a"} Sep 17 17:35:18.061925 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:35:18.061877 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lbj98" podStartSLOduration=253.288292627 podStartE2EDuration="4m14.061862458s" totalImagesPullingTime="773.569831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:18.060211068 +0000 UTC m=+254.622375929" watchObservedRunningTime="2026-09-17 17:35:18.061862458 +0000 UTC m=+254.624027317" Sep 17 17:36:03.861571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:03.861546 2566 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 17:36:49.951102 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.951069 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc"] Sep 17 17:36:49.951532 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.951328 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" containerName="registry" Sep 17 17:36:49.951532 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.951338 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" containerName="registry" Sep 17 17:36:49.951532 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.951383 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f1ae7f-7b01-4378-9f18-77df3071c780" containerName="registry" Sep 17 17:36:49.953999 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.953984 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:49.956866 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.956841 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 17:36:49.957144 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.957129 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 17:36:49.957875 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.957854 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mxtr7\"" Sep 17 17:36:49.965384 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:49.965362 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc"] Sep 17 17:36:50.089589 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.089556 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adf73672-e7bc-4363-913b-6d3d07498409-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5vmvc\" (UID: \"adf73672-e7bc-4363-913b-6d3d07498409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.089783 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.089601 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdj2z\" (UniqueName: \"kubernetes.io/projected/adf73672-e7bc-4363-913b-6d3d07498409-kube-api-access-jdj2z\") pod \"cert-manager-cainjector-559cdd8c87-5vmvc\" (UID: \"adf73672-e7bc-4363-913b-6d3d07498409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.190176 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.190146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adf73672-e7bc-4363-913b-6d3d07498409-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5vmvc\" (UID: \"adf73672-e7bc-4363-913b-6d3d07498409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.190370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.190185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdj2z\" (UniqueName: \"kubernetes.io/projected/adf73672-e7bc-4363-913b-6d3d07498409-kube-api-access-jdj2z\") pod \"cert-manager-cainjector-559cdd8c87-5vmvc\" (UID: \"adf73672-e7bc-4363-913b-6d3d07498409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.201393 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.201333 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adf73672-e7bc-4363-913b-6d3d07498409-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5vmvc\" (UID: \"adf73672-e7bc-4363-913b-6d3d07498409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.202101 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.202082 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdj2z\" (UniqueName: \"kubernetes.io/projected/adf73672-e7bc-4363-913b-6d3d07498409-kube-api-access-jdj2z\") pod \"cert-manager-cainjector-559cdd8c87-5vmvc\" (UID: \"adf73672-e7bc-4363-913b-6d3d07498409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.262880 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.262851 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" Sep 17 17:36:50.528884 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.528815 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc"] Sep 17 17:36:50.531766 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:36:50.531740 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadf73672_e7bc_4363_913b_6d3d07498409.slice/crio-8db8891603ee1ac4eb11e7fd81f0bb3394b69922187353d6c1af0ca8353fbdbe WatchSource:0}: Error finding container 8db8891603ee1ac4eb11e7fd81f0bb3394b69922187353d6c1af0ca8353fbdbe: Status 404 returned error can't find the container with id 8db8891603ee1ac4eb11e7fd81f0bb3394b69922187353d6c1af0ca8353fbdbe Sep 17 17:36:50.533719 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:50.533702 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:36:51.343538 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:51.343506 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" event={"ID":"adf73672-e7bc-4363-913b-6d3d07498409","Type":"ContainerStarted","Data":"8db8891603ee1ac4eb11e7fd81f0bb3394b69922187353d6c1af0ca8353fbdbe"} Sep 17 17:36:54.355031 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:54.354997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" event={"ID":"adf73672-e7bc-4363-913b-6d3d07498409","Type":"ContainerStarted","Data":"116bfb871ae2d84917caad230bc6b00444a8690603800199b8d9f6db98d48181"} Sep 17 17:36:54.378177 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:36:54.378130 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5vmvc" podStartSLOduration=2.472445233 podStartE2EDuration="5.378114645s" totalImagesPullingTime="2.905669412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:54.376446812 +0000 UTC m=+350.938611671" watchObservedRunningTime="2026-09-17 17:36:54.378114645 +0000 UTC m=+350.940279506" Sep 17 17:37:15.029946 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.029866 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm"] Sep 17 17:37:15.033480 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.033465 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.039887 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.039756 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 17:37:15.039887 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.039771 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 17:37:15.039887 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.039838 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jt585\"" Sep 17 17:37:15.043405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.043386 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 17:37:15.043607 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.043592 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 17:37:15.053769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.053748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqp4w\" (UniqueName: \"kubernetes.io/projected/34a1d4cf-3914-42ab-bfbf-93e21ced2168-kube-api-access-lqp4w\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.053844 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.053787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34a1d4cf-3914-42ab-bfbf-93e21ced2168-webhook-cert\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.053844 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.053833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34a1d4cf-3914-42ab-bfbf-93e21ced2168-apiservice-cert\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.053910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.053856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34a1d4cf-3914-42ab-bfbf-93e21ced2168-tmp\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.053910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.053883 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34a1d4cf-3914-42ab-bfbf-93e21ced2168-manifests\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.081076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.081039 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm"] Sep 17 17:37:15.154414 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.154377 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34a1d4cf-3914-42ab-bfbf-93e21ced2168-webhook-cert\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.154573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.154425 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34a1d4cf-3914-42ab-bfbf-93e21ced2168-apiservice-cert\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.154628 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.154586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34a1d4cf-3914-42ab-bfbf-93e21ced2168-tmp\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.154682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.154629 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34a1d4cf-3914-42ab-bfbf-93e21ced2168-manifests\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.154779 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.154751 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqp4w\" (UniqueName: \"kubernetes.io/projected/34a1d4cf-3914-42ab-bfbf-93e21ced2168-kube-api-access-lqp4w\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.154927 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.154907 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34a1d4cf-3914-42ab-bfbf-93e21ced2168-manifests\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.156764 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.156738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34a1d4cf-3914-42ab-bfbf-93e21ced2168-tmp\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.156883 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.156868 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34a1d4cf-3914-42ab-bfbf-93e21ced2168-apiservice-cert\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.156973 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.156939 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34a1d4cf-3914-42ab-bfbf-93e21ced2168-webhook-cert\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.169068 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.169050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqp4w\" (UniqueName: \"kubernetes.io/projected/34a1d4cf-3914-42ab-bfbf-93e21ced2168-kube-api-access-lqp4w\") pod \"opendatahub-operator-controller-manager-584f5b8894-qvzwm\" (UID: \"34a1d4cf-3914-42ab-bfbf-93e21ced2168\") " pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.342507 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.342488 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:15.624887 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:15.624732 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm"] Sep 17 17:37:15.627558 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:37:15.627523 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a1d4cf_3914_42ab_bfbf_93e21ced2168.slice/crio-af36810be6256307e57d6ed1505c8ec78a7eff06d3c28f66e2f61cdade0a0f38 WatchSource:0}: Error finding container af36810be6256307e57d6ed1505c8ec78a7eff06d3c28f66e2f61cdade0a0f38: Status 404 returned error can't find the container with id af36810be6256307e57d6ed1505c8ec78a7eff06d3c28f66e2f61cdade0a0f38 Sep 17 17:37:16.429530 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:16.429490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" event={"ID":"34a1d4cf-3914-42ab-bfbf-93e21ced2168","Type":"ContainerStarted","Data":"af36810be6256307e57d6ed1505c8ec78a7eff06d3c28f66e2f61cdade0a0f38"} Sep 17 17:37:19.444490 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:19.444453 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" event={"ID":"34a1d4cf-3914-42ab-bfbf-93e21ced2168","Type":"ContainerStarted","Data":"7bec581627c10f755f3b4110efe941d164161093807a48a647967698efa28343"} Sep 17 17:37:20.449023 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:20.448994 2566 generic.go:356] "Generic (PLEG): container finished" podID="34a1d4cf-3914-42ab-bfbf-93e21ced2168" containerID="7bec581627c10f755f3b4110efe941d164161093807a48a647967698efa28343" exitCode=0 Sep 17 17:37:20.449415 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:20.449081 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" event={"ID":"34a1d4cf-3914-42ab-bfbf-93e21ced2168","Type":"ContainerDied","Data":"7bec581627c10f755f3b4110efe941d164161093807a48a647967698efa28343"} Sep 17 17:37:21.454926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:21.454892 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" event={"ID":"34a1d4cf-3914-42ab-bfbf-93e21ced2168","Type":"ContainerStarted","Data":"dca08dac854d7589ada3da21098ea342375af47b771b5c6c4eb8b365f88ceb50"} Sep 17 17:37:21.455359 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:21.454990 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:21.482641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:21.482595 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" podStartSLOduration=2.868896937 podStartE2EDuration="7.482579388s" totalImagesPullingTime="3.55141779s" totalInitContainerRuntime="1.062264661s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:21.482237498 +0000 UTC m=+378.044402358" watchObservedRunningTime="2026-09-17 17:37:21.482579388 +0000 UTC m=+378.044744247" Sep 17 17:37:32.460307 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:32.460271 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-584f5b8894-qvzwm" Sep 17 17:37:34.798734 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.798698 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5499794fdf-nt54m"] Sep 17 17:37:34.801520 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.801504 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.804282 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.804257 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 17:37:34.804411 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.804261 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 17:37:34.804411 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.804266 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zb8d4\"" Sep 17 17:37:34.804411 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.804353 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 17:37:34.805926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.805910 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 17:37:34.820473 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.820446 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5499794fdf-nt54m"] Sep 17 17:37:34.891722 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.891695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0489bd-a206-419b-88eb-8de14a1c9595-tmp\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.891824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.891734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qk2\" (UniqueName: \"kubernetes.io/projected/7c0489bd-a206-419b-88eb-8de14a1c9595-kube-api-access-j6qk2\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.891824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.891794 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0489bd-a206-419b-88eb-8de14a1c9595-tls-certs\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.992072 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.992048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0489bd-a206-419b-88eb-8de14a1c9595-tmp\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.992231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.992086 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qk2\" (UniqueName: \"kubernetes.io/projected/7c0489bd-a206-419b-88eb-8de14a1c9595-kube-api-access-j6qk2\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.992231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.992116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0489bd-a206-419b-88eb-8de14a1c9595-tls-certs\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.994333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.994310 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0489bd-a206-419b-88eb-8de14a1c9595-tmp\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:34.994489 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:34.994474 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0489bd-a206-419b-88eb-8de14a1c9595-tls-certs\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:35.009626 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:35.009595 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qk2\" (UniqueName: \"kubernetes.io/projected/7c0489bd-a206-419b-88eb-8de14a1c9595-kube-api-access-j6qk2\") pod \"kube-auth-proxy-5499794fdf-nt54m\" (UID: \"7c0489bd-a206-419b-88eb-8de14a1c9595\") " pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:35.110584 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:35.110559 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" Sep 17 17:37:35.391406 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:35.391381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5499794fdf-nt54m"] Sep 17 17:37:35.393075 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:37:35.393046 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0489bd_a206_419b_88eb_8de14a1c9595.slice/crio-2a982c3862e16e6c49ca9aaf08a90f83e30f62b5c7e49df36d374290b6b4a0d8 WatchSource:0}: Error finding container 2a982c3862e16e6c49ca9aaf08a90f83e30f62b5c7e49df36d374290b6b4a0d8: Status 404 returned error can't find the container with id 2a982c3862e16e6c49ca9aaf08a90f83e30f62b5c7e49df36d374290b6b4a0d8 Sep 17 17:37:35.504454 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:35.504424 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" event={"ID":"7c0489bd-a206-419b-88eb-8de14a1c9595","Type":"ContainerStarted","Data":"2a982c3862e16e6c49ca9aaf08a90f83e30f62b5c7e49df36d374290b6b4a0d8"} Sep 17 17:37:36.509690 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.509656 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6"] Sep 17 17:37:36.512892 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.512871 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.517070 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.517047 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 17 17:37:36.517275 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.517253 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dllt\"" Sep 17 17:37:36.517360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.517277 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 17:37:36.517360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.517316 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 17 17:37:36.517695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.517674 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 17 17:37:36.532704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.532681 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6"] Sep 17 17:37:36.603140 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603113 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb7de0d-068e-4240-a069-73645cf79f98-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.603140 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603143 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.603337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.603337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.603337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603274 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.603337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.603337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.603328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctf5t\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-kube-api-access-ctf5t\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704621 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctf5t\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-kube-api-access-ctf5t\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb7de0d-068e-4240-a069-73645cf79f98-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.704842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.704735 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.705338 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.705308 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.707530 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.707490 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb7de0d-068e-4240-a069-73645cf79f98-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.707655 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.707547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.707714 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.707677 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.708173 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.708156 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.715442 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.715389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.717470 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.717451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctf5t\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-kube-api-access-ctf5t\") pod \"istiod-openshift-gateway-7b7995d5cd-zfzw6\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:36.823485 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:36.823456 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:37.115325 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:37.114839 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6"] Sep 17 17:37:38.254219 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:37:38.254181 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb7de0d_068e_4240_a069_73645cf79f98.slice/crio-8a4ffdf77e10ecac0f3cf19788584c981584450063ca3677e0832c92277d63f2 WatchSource:0}: Error finding container 8a4ffdf77e10ecac0f3cf19788584c981584450063ca3677e0832c92277d63f2: Status 404 returned error can't find the container with id 8a4ffdf77e10ecac0f3cf19788584c981584450063ca3677e0832c92277d63f2 Sep 17 17:37:38.518151 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:38.518063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" event={"ID":"7c0489bd-a206-419b-88eb-8de14a1c9595","Type":"ContainerStarted","Data":"262d28cc0426ceaf191ddfc3c6d7325a59be2d05c8fc18d5c5ea8a8fd5443c54"} Sep 17 17:37:38.519236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:38.519212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" event={"ID":"7eb7de0d-068e-4240-a069-73645cf79f98","Type":"ContainerStarted","Data":"8a4ffdf77e10ecac0f3cf19788584c981584450063ca3677e0832c92277d63f2"} Sep 17 17:37:38.537838 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:38.537794 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5499794fdf-nt54m" podStartSLOduration=1.626787239 podStartE2EDuration="4.537780795s" totalImagesPullingTime="2.910993556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:38.537036762 +0000 UTC m=+395.099201633" watchObservedRunningTime="2026-09-17 17:37:38.537780795 +0000 UTC m=+395.099945655" Sep 17 17:37:40.663170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:40.663129 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 17:37:40.663463 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:40.663207 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 17:37:41.531555 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:41.531522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" event={"ID":"7eb7de0d-068e-4240-a069-73645cf79f98","Type":"ContainerStarted","Data":"c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d"} Sep 17 17:37:41.531694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:41.531681 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:41.532997 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:41.532976 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:41.574140 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:41.574098 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" podStartSLOduration=3.167850278 podStartE2EDuration="5.574084737s" totalImagesPullingTime="2.406234459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:41.57375321 +0000 UTC m=+398.135918096" watchObservedRunningTime="2026-09-17 17:37:41.574084737 +0000 UTC m=+398.136249596" Sep 17 17:37:46.021256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.021217 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n9ptj"] Sep 17 17:37:46.024466 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.024442 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.026873 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.026852 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 17 17:37:46.027027 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.027008 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-d9xxp\"" Sep 17 17:37:46.037404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.037381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n9ptj"] Sep 17 17:37:46.073824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.073798 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-tls-certs\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.073925 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.073833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxx5\" (UniqueName: \"kubernetes.io/projected/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-kube-api-access-nbxx5\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.175167 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.175135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-tls-certs\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.175363 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.175178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxx5\" (UniqueName: \"kubernetes.io/projected/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-kube-api-access-nbxx5\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.175363 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:37:46.175289 2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 17 17:37:46.175363 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:37:46.175358 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-tls-certs podName:1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:46.675341166 +0000 UTC m=+403.237506016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-tls-certs") pod "model-serving-api-54d89bd5fc-n9ptj" (UID: "1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4") : secret "model-serving-api-tls" not found Sep 17 17:37:46.184804 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.184776 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxx5\" (UniqueName: \"kubernetes.io/projected/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-kube-api-access-nbxx5\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.679822 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.679778 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-tls-certs\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.682295 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.682274 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4-tls-certs\") pod \"model-serving-api-54d89bd5fc-n9ptj\" (UID: \"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4\") " pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:46.934823 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:46.934744 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:47.208160 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:47.208091 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n9ptj"] Sep 17 17:37:47.211165 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:37:47.211140 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1d7ab9_7f46_4896_b9f4_97ed05a5ecd4.slice/crio-2bb3c1fb9c3024748336436884021b0d1078af567bac7ee315507fd017298e68 WatchSource:0}: Error finding container 2bb3c1fb9c3024748336436884021b0d1078af567bac7ee315507fd017298e68: Status 404 returned error can't find the container with id 2bb3c1fb9c3024748336436884021b0d1078af567bac7ee315507fd017298e68 Sep 17 17:37:47.553227 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:47.553196 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" event={"ID":"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4","Type":"ContainerStarted","Data":"2bb3c1fb9c3024748336436884021b0d1078af567bac7ee315507fd017298e68"} Sep 17 17:37:49.562095 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:49.562054 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" event={"ID":"1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4","Type":"ContainerStarted","Data":"8dafc081e84d8e547ab0f93a483e657ec11ce20c87a4af7f60436ca4fa212d7f"} Sep 17 17:37:49.562516 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:49.562200 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:37:49.581944 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:49.581880 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" podStartSLOduration=2.009281798 podStartE2EDuration="3.581856004s" totalImagesPullingTime="1.572574206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:49.580620291 +0000 UTC m=+406.142785155" watchObservedRunningTime="2026-09-17 17:37:49.581856004 +0000 UTC m=+406.144020864" Sep 17 17:37:55.670404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.670370 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dxldr"] Sep 17 17:37:55.678205 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.678182 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:55.683431 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.683407 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 17 17:37:55.687432 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.687411 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rqxkz\"" Sep 17 17:37:55.692671 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.692649 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dxldr"] Sep 17 17:37:55.694886 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.694857 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c"] Sep 17 17:37:55.697580 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.697561 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.704273 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.704245 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 17 17:37:55.704516 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.704502 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vwhfk\"" Sep 17 17:37:55.715937 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.715906 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c"] Sep 17 17:37:55.743636 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.743605 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6zs\" (UniqueName: \"kubernetes.io/projected/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-kube-api-access-qp6zs\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:55.743745 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.743659 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:55.743792 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.743762 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65dd8a4a-3c8b-4a84-856e-902773c15f8c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j7p4c\" (UID: \"65dd8a4a-3c8b-4a84-856e-902773c15f8c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.743827 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.743788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xnxq\" (UniqueName: \"kubernetes.io/projected/65dd8a4a-3c8b-4a84-856e-902773c15f8c-kube-api-access-9xnxq\") pod \"llmisvc-controller-manager-777c6bd7d5-j7p4c\" (UID: \"65dd8a4a-3c8b-4a84-856e-902773c15f8c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.844916 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.844889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65dd8a4a-3c8b-4a84-856e-902773c15f8c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j7p4c\" (UID: \"65dd8a4a-3c8b-4a84-856e-902773c15f8c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.844916 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.844920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xnxq\" (UniqueName: \"kubernetes.io/projected/65dd8a4a-3c8b-4a84-856e-902773c15f8c-kube-api-access-9xnxq\") pod \"llmisvc-controller-manager-777c6bd7d5-j7p4c\" (UID: \"65dd8a4a-3c8b-4a84-856e-902773c15f8c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.845084 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.844942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6zs\" (UniqueName: \"kubernetes.io/projected/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-kube-api-access-qp6zs\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:55.845084 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.844979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:55.845084 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:37:55.845076 2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 17 17:37:55.845187 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:37:55.845132 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-cert podName:0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c nodeName:}" failed. No retries permitted until 2026-09-17 17:37:56.3451132 +0000 UTC m=+412.907278057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-cert") pod "kserve-controller-manager-d6f4b65b8-dxldr" (UID: "0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c") : secret "kserve-webhook-server-cert" not found Sep 17 17:37:55.847377 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.847358 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65dd8a4a-3c8b-4a84-856e-902773c15f8c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j7p4c\" (UID: \"65dd8a4a-3c8b-4a84-856e-902773c15f8c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.861417 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.861389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xnxq\" (UniqueName: \"kubernetes.io/projected/65dd8a4a-3c8b-4a84-856e-902773c15f8c-kube-api-access-9xnxq\") pod \"llmisvc-controller-manager-777c6bd7d5-j7p4c\" (UID: \"65dd8a4a-3c8b-4a84-856e-902773c15f8c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:55.863371 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:55.863347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6zs\" (UniqueName: \"kubernetes.io/projected/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-kube-api-access-qp6zs\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:56.006847 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.006762 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:37:56.286444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.286417 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c"] Sep 17 17:37:56.288636 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:37:56.288611 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65dd8a4a_3c8b_4a84_856e_902773c15f8c.slice/crio-24b7a357c04b74c1d2193dfbf8ed5157aa89eecb57d363b6af251236c77d791f WatchSource:0}: Error finding container 24b7a357c04b74c1d2193dfbf8ed5157aa89eecb57d363b6af251236c77d791f: Status 404 returned error can't find the container with id 24b7a357c04b74c1d2193dfbf8ed5157aa89eecb57d363b6af251236c77d791f Sep 17 17:37:56.348301 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.348268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:56.350463 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.350445 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c-cert\") pod \"kserve-controller-manager-d6f4b65b8-dxldr\" (UID: \"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:56.587740 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.587712 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:37:56.588800 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.588773 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" event={"ID":"65dd8a4a-3c8b-4a84-856e-902773c15f8c","Type":"ContainerStarted","Data":"24b7a357c04b74c1d2193dfbf8ed5157aa89eecb57d363b6af251236c77d791f"} Sep 17 17:37:56.876883 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:56.876817 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dxldr"] Sep 17 17:37:56.879786 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:37:56.879763 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c91bdae_c4aa_41ab_9dca_5a4fcb09de9c.slice/crio-0abbb57beb261363ad4dda119a909d1b2257ed8124fb60373d01fb5249e39dec WatchSource:0}: Error finding container 0abbb57beb261363ad4dda119a909d1b2257ed8124fb60373d01fb5249e39dec: Status 404 returned error can't find the container with id 0abbb57beb261363ad4dda119a909d1b2257ed8124fb60373d01fb5249e39dec Sep 17 17:37:57.594725 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:57.594694 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" event={"ID":"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c","Type":"ContainerStarted","Data":"0abbb57beb261363ad4dda119a909d1b2257ed8124fb60373d01fb5249e39dec"} Sep 17 17:37:59.675796 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.675768 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6"] Sep 17 17:37:59.676192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.676022 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" podUID="7eb7de0d-068e-4240-a069-73645cf79f98" containerName="discovery" containerID="cri-o://c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d" gracePeriod=30 Sep 17 17:37:59.917138 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.917108 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:37:59.978896 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.978863 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb7de0d-068e-4240-a069-73645cf79f98-local-certs\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.978896 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.978897 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-kubeconfig\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.979139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.978946 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-cacerts\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.979139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.978995 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctf5t\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-kube-api-access-ctf5t\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.979139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.979019 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-istio-token\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.979139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.979056 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-dns-cert\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.979139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.979082 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-ca-configmap\") pod \"7eb7de0d-068e-4240-a069-73645cf79f98\" (UID: \"7eb7de0d-068e-4240-a069-73645cf79f98\") " Sep 17 17:37:59.979610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.979584 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:37:59.981304 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.981272 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb7de0d-068e-4240-a069-73645cf79f98-local-certs" (OuterVolumeSpecName: "local-certs") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:37:59.981397 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.981296 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-kube-api-access-ctf5t" (OuterVolumeSpecName: "kube-api-access-ctf5t") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "kube-api-access-ctf5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:37:59.981465 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.981446 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-istio-token" (OuterVolumeSpecName: "istio-token") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:37:59.981502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.981460 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:37:59.981615 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.981589 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-cacerts" (OuterVolumeSpecName: "cacerts") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:37:59.981711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:37:59.981611 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7eb7de0d-068e-4240-a069-73645cf79f98" (UID: "7eb7de0d-068e-4240-a069-73645cf79f98"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:38:00.079695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079659 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-cacerts\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.079695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079690 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctf5t\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-kube-api-access-ctf5t\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.079695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079702 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb7de0d-068e-4240-a069-73645cf79f98-istio-token\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.079914 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079711 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-dns-cert\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.079914 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079720 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb7de0d-068e-4240-a069-73645cf79f98-istio-csr-ca-configmap\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.079914 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079728 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb7de0d-068e-4240-a069-73645cf79f98-local-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.079914 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.079736 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb7de0d-068e-4240-a069-73645cf79f98-istio-kubeconfig\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:38:00.571136 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.571109 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-n9ptj" Sep 17 17:38:00.610687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.610061 2566 generic.go:356] "Generic (PLEG): container finished" podID="7eb7de0d-068e-4240-a069-73645cf79f98" containerID="c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d" exitCode=0 Sep 17 17:38:00.610687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.610177 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" event={"ID":"7eb7de0d-068e-4240-a069-73645cf79f98","Type":"ContainerDied","Data":"c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d"} Sep 17 17:38:00.610687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.610254 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" event={"ID":"7eb7de0d-068e-4240-a069-73645cf79f98","Type":"ContainerDied","Data":"8a4ffdf77e10ecac0f3cf19788584c981584450063ca3677e0832c92277d63f2"} Sep 17 17:38:00.610687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.610290 2566 scope.go:117] "RemoveContainer" containerID="c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d" Sep 17 17:38:00.611000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.610856 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6" Sep 17 17:38:00.613027 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.612977 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" event={"ID":"65dd8a4a-3c8b-4a84-856e-902773c15f8c","Type":"ContainerStarted","Data":"98d404b11fb5fdd9069c661ea87ba32e999a17c830c9dbf941ba54ce57166f29"} Sep 17 17:38:00.613241 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.613215 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:38:00.616695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.616656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" event={"ID":"0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c","Type":"ContainerStarted","Data":"b81f7b1ab5bb0d1ded57d11a972e4bc941298cf8b09f1debaa4541074cc24e51"} Sep 17 17:38:00.616798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.616748 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:38:00.623480 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.623453 2566 scope.go:117] "RemoveContainer" containerID="c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d" Sep 17 17:38:00.623746 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:38:00.623726 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d\": container with ID starting with c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d not found: ID does not exist" containerID="c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d" Sep 17 17:38:00.623804 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.623753 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d"} err="failed to get container status \"c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d\": rpc error: code = NotFound desc = could not find container \"c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d\": container with ID starting with c679910e0b7d3699a3123f7aa0203422fcd0bdd6a48e250deb574faa53390f2d not found: ID does not exist" Sep 17 17:38:00.640000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.639921 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" podStartSLOduration=2.925378096 podStartE2EDuration="5.639903416s" totalImagesPullingTime="2.71452532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:00.63988412 +0000 UTC m=+417.202048983" watchObservedRunningTime="2026-09-17 17:38:00.639903416 +0000 UTC m=+417.202068277" Sep 17 17:38:00.672051 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.671999 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" podStartSLOduration=2.367880833 podStartE2EDuration="5.671982598s" totalImagesPullingTime="3.304101765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:00.670005984 +0000 UTC m=+417.232170844" watchObservedRunningTime="2026-09-17 17:38:00.671982598 +0000 UTC m=+417.234147459" Sep 17 17:38:00.699622 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.699570 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6"] Sep 17 17:38:00.706488 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:00.706463 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zfzw6"] Sep 17 17:38:02.148250 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:02.148221 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb7de0d-068e-4240-a069-73645cf79f98" path="/var/lib/kubelet/pods/7eb7de0d-068e-4240-a069-73645cf79f98/volumes" Sep 17 17:38:31.623174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:31.623142 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j7p4c" Sep 17 17:38:31.626313 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:38:31.626291 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dxldr" Sep 17 17:39:02.540408 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.540368 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5"] Sep 17 17:39:02.540836 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.540668 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb7de0d-068e-4240-a069-73645cf79f98" containerName="discovery" Sep 17 17:39:02.540836 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.540679 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb7de0d-068e-4240-a069-73645cf79f98" containerName="discovery" Sep 17 17:39:02.540836 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.540722 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb7de0d-068e-4240-a069-73645cf79f98" containerName="discovery" Sep 17 17:39:02.543576 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.543560 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:02.546256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.546231 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 17:39:02.546404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.546303 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 17:39:02.547240 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.547223 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2dlvl\"" Sep 17 17:39:02.547310 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.547223 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 17 17:39:02.554724 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.554701 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5"] Sep 17 17:39:02.597397 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.597350 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hks\" (UniqueName: \"kubernetes.io/projected/31051d3c-f712-49f7-b929-af039212ce93-kube-api-access-82hks\") pod \"dns-operator-controller-manager-7b84cd7c9c-zgws5\" (UID: \"31051d3c-f712-49f7-b929-af039212ce93\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:02.698687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.698642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hks\" (UniqueName: \"kubernetes.io/projected/31051d3c-f712-49f7-b929-af039212ce93-kube-api-access-82hks\") pod \"dns-operator-controller-manager-7b84cd7c9c-zgws5\" (UID: \"31051d3c-f712-49f7-b929-af039212ce93\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:02.718446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.718420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hks\" (UniqueName: \"kubernetes.io/projected/31051d3c-f712-49f7-b929-af039212ce93-kube-api-access-82hks\") pod \"dns-operator-controller-manager-7b84cd7c9c-zgws5\" (UID: \"31051d3c-f712-49f7-b929-af039212ce93\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:02.853238 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:02.853207 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:03.133343 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:03.133319 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5"] Sep 17 17:39:03.135784 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:03.135747 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31051d3c_f712_49f7_b929_af039212ce93.slice/crio-ebab096cfffbbbfd124c73bda361a06403f823a2c013aae2bcdad964d849744d WatchSource:0}: Error finding container ebab096cfffbbbfd124c73bda361a06403f823a2c013aae2bcdad964d849744d: Status 404 returned error can't find the container with id ebab096cfffbbbfd124c73bda361a06403f823a2c013aae2bcdad964d849744d Sep 17 17:39:03.838185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:03.838144 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" event={"ID":"31051d3c-f712-49f7-b929-af039212ce93","Type":"ContainerStarted","Data":"ebab096cfffbbbfd124c73bda361a06403f823a2c013aae2bcdad964d849744d"} Sep 17 17:39:05.936587 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:05.936565 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 17 17:39:06.853503 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:06.853470 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" event={"ID":"31051d3c-f712-49f7-b929-af039212ce93","Type":"ContainerStarted","Data":"ee088e107c971db2ef299ccd725364a2df237e0ea1cd5f7b61e47bdf6eeaeba1"} Sep 17 17:39:06.853698 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:06.853595 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:07.163503 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.163403 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" podStartSLOduration=2.36828872 podStartE2EDuration="5.163385628s" totalImagesPullingTime="2.795096908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:06.887418067 +0000 UTC m=+483.449582926" watchObservedRunningTime="2026-09-17 17:39:07.163385628 +0000 UTC m=+483.725550487" Sep 17 17:39:07.164434 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.164416 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb"] Sep 17 17:39:07.167493 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.167472 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.172918 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.172894 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wwhmw\"" Sep 17 17:39:07.199107 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.199085 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb"] Sep 17 17:39:07.234022 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.233991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0717d155-00d8-458e-afda-5be86f209550-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.234149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.234043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjqn\" (UniqueName: \"kubernetes.io/projected/0717d155-00d8-458e-afda-5be86f209550-kube-api-access-4kjqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.335300 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.335272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0717d155-00d8-458e-afda-5be86f209550-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.335404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.335327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjqn\" (UniqueName: \"kubernetes.io/projected/0717d155-00d8-458e-afda-5be86f209550-kube-api-access-4kjqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.335728 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.335711 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0717d155-00d8-458e-afda-5be86f209550-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.359536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.359510 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjqn\" (UniqueName: \"kubernetes.io/projected/0717d155-00d8-458e-afda-5be86f209550-kube-api-access-4kjqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.477116 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.477046 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:07.765780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.765711 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb"] Sep 17 17:39:07.768901 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:07.768871 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0717d155_00d8_458e_afda_5be86f209550.slice/crio-50fde20066d9097a7b4bb7b255b30c9b4d6a83f741996cdaf98ee7bba0c27e05 WatchSource:0}: Error finding container 50fde20066d9097a7b4bb7b255b30c9b4d6a83f741996cdaf98ee7bba0c27e05: Status 404 returned error can't find the container with id 50fde20066d9097a7b4bb7b255b30c9b4d6a83f741996cdaf98ee7bba0c27e05 Sep 17 17:39:07.858929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:07.858899 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" event={"ID":"0717d155-00d8-458e-afda-5be86f209550","Type":"ContainerStarted","Data":"50fde20066d9097a7b4bb7b255b30c9b4d6a83f741996cdaf98ee7bba0c27e05"} Sep 17 17:39:09.601332 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.601297 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-z65qv"] Sep 17 17:39:09.604314 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.604298 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:09.611061 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.611042 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-h8xql\"" Sep 17 17:39:09.624921 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.624900 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-z65qv"] Sep 17 17:39:09.650030 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.650004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4msvq\" (UniqueName: \"kubernetes.io/projected/3d36aa81-6545-4d9b-98e5-babd8336ea87-kube-api-access-4msvq\") pod \"authorino-operator-76b478bd78-z65qv\" (UID: \"3d36aa81-6545-4d9b-98e5-babd8336ea87\") " pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:09.751018 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.750988 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4msvq\" (UniqueName: \"kubernetes.io/projected/3d36aa81-6545-4d9b-98e5-babd8336ea87-kube-api-access-4msvq\") pod \"authorino-operator-76b478bd78-z65qv\" (UID: \"3d36aa81-6545-4d9b-98e5-babd8336ea87\") " pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:09.760109 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.760089 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4msvq\" (UniqueName: \"kubernetes.io/projected/3d36aa81-6545-4d9b-98e5-babd8336ea87-kube-api-access-4msvq\") pod \"authorino-operator-76b478bd78-z65qv\" (UID: \"3d36aa81-6545-4d9b-98e5-babd8336ea87\") " pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:09.914012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:09.913916 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:10.209770 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:10.209744 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-z65qv"] Sep 17 17:39:10.212043 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:10.212008 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d36aa81_6545_4d9b_98e5_babd8336ea87.slice/crio-e364e43d858dcfbf00be05fc23c3b86a1b754aa1f62d4eda0706b74c2320165d WatchSource:0}: Error finding container e364e43d858dcfbf00be05fc23c3b86a1b754aa1f62d4eda0706b74c2320165d: Status 404 returned error can't find the container with id e364e43d858dcfbf00be05fc23c3b86a1b754aa1f62d4eda0706b74c2320165d Sep 17 17:39:10.873182 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:10.873145 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" event={"ID":"3d36aa81-6545-4d9b-98e5-babd8336ea87","Type":"ContainerStarted","Data":"e364e43d858dcfbf00be05fc23c3b86a1b754aa1f62d4eda0706b74c2320165d"} Sep 17 17:39:11.773428 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.773395 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh"] Sep 17 17:39:11.776849 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.776829 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:11.779424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.779401 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qnk4z\"" Sep 17 17:39:11.796711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.796692 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh"] Sep 17 17:39:11.869734 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.869700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gghbd\" (UniqueName: \"kubernetes.io/projected/2d15bb72-01cf-48db-b882-7dd9a5e31dba-kube-api-access-gghbd\") pod \"limitador-operator-controller-manager-74495c69f8-cvxrh\" (UID: \"2d15bb72-01cf-48db-b882-7dd9a5e31dba\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:11.970135 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.970105 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghbd\" (UniqueName: \"kubernetes.io/projected/2d15bb72-01cf-48db-b882-7dd9a5e31dba-kube-api-access-gghbd\") pod \"limitador-operator-controller-manager-74495c69f8-cvxrh\" (UID: \"2d15bb72-01cf-48db-b882-7dd9a5e31dba\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:11.982478 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:11.982451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghbd\" (UniqueName: \"kubernetes.io/projected/2d15bb72-01cf-48db-b882-7dd9a5e31dba-kube-api-access-gghbd\") pod \"limitador-operator-controller-manager-74495c69f8-cvxrh\" (UID: \"2d15bb72-01cf-48db-b882-7dd9a5e31dba\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:12.089444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:12.089410 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:12.529821 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:12.529791 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh"] Sep 17 17:39:12.533098 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:12.533067 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d15bb72_01cf_48db_b882_7dd9a5e31dba.slice/crio-cba6309d4bdf897c59584af5a12d56815e0ae6463b5bf5cd56446f3d29fac5e3 WatchSource:0}: Error finding container cba6309d4bdf897c59584af5a12d56815e0ae6463b5bf5cd56446f3d29fac5e3: Status 404 returned error can't find the container with id cba6309d4bdf897c59584af5a12d56815e0ae6463b5bf5cd56446f3d29fac5e3 Sep 17 17:39:12.885038 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:12.884998 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" event={"ID":"2d15bb72-01cf-48db-b882-7dd9a5e31dba","Type":"ContainerStarted","Data":"cba6309d4bdf897c59584af5a12d56815e0ae6463b5bf5cd56446f3d29fac5e3"} Sep 17 17:39:12.887014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:12.886980 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" event={"ID":"0717d155-00d8-458e-afda-5be86f209550","Type":"ContainerStarted","Data":"e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452"} Sep 17 17:39:12.887135 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:12.887112 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:12.910362 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:12.910307 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" podStartSLOduration=1.347439798 podStartE2EDuration="5.91029305s" totalImagesPullingTime="4.562853252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:12.9083627 +0000 UTC m=+489.470527586" watchObservedRunningTime="2026-09-17 17:39:12.91029305 +0000 UTC m=+489.472457909" Sep 17 17:39:13.892218 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:13.892180 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" event={"ID":"3d36aa81-6545-4d9b-98e5-babd8336ea87","Type":"ContainerStarted","Data":"52efe0395da0c0d018bb1d088b15f9a66156b6a7dfb7c8af694ac3ea2b514e3e"} Sep 17 17:39:13.892609 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:13.892273 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:13.912862 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:13.912816 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" podStartSLOduration=2.121973849 podStartE2EDuration="4.912802589s" totalImagesPullingTime="2.79082874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:13.912145314 +0000 UTC m=+490.474310175" watchObservedRunningTime="2026-09-17 17:39:13.912802589 +0000 UTC m=+490.474967448" Sep 17 17:39:17.860941 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:17.860913 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-zgws5" Sep 17 17:39:23.894980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:23.894893 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:23.943706 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:23.943674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" event={"ID":"2d15bb72-01cf-48db-b882-7dd9a5e31dba","Type":"ContainerStarted","Data":"9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941"} Sep 17 17:39:23.943890 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:23.943756 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:23.973307 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:23.973261 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" podStartSLOduration=1.866706846 podStartE2EDuration="12.973243421s" totalImagesPullingTime="11.106536575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:23.971301607 +0000 UTC m=+500.533466467" watchObservedRunningTime="2026-09-17 17:39:23.973243421 +0000 UTC m=+500.535408283" Sep 17 17:39:24.898032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:24.898001 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-z65qv" Sep 17 17:39:25.760988 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.760943 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm"] Sep 17 17:39:25.764187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.764166 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:25.795859 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.795835 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm"] Sep 17 17:39:25.872165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.872138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:25.872278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.872207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89q9\" (UniqueName: \"kubernetes.io/projected/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-kube-api-access-f89q9\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:25.973493 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.973468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:25.973817 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.973511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89q9\" (UniqueName: \"kubernetes.io/projected/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-kube-api-access-f89q9\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:25.973854 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.973818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:25.984625 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:25.984593 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89q9\" (UniqueName: \"kubernetes.io/projected/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-kube-api-access-f89q9\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.073816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.073794 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.221379 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.221348 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb"] Sep 17 17:39:26.221528 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.221498 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" podUID="0717d155-00d8-458e-afda-5be86f209550" containerName="manager" containerID="cri-o://e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452" gracePeriod=2 Sep 17 17:39:26.237171 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.237142 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb"] Sep 17 17:39:26.240469 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.240437 2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0717d155-00d8-458e-afda-5be86f209550\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\": pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.278092 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.278070 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm"] Sep 17 17:39:26.292790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.292770 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm"] Sep 17 17:39:26.302043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.302023 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67"] Sep 17 17:39:26.302287 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.302276 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0717d155-00d8-458e-afda-5be86f209550" containerName="manager" Sep 17 17:39:26.302331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.302288 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0717d155-00d8-458e-afda-5be86f209550" containerName="manager" Sep 17 17:39:26.302364 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.302340 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0717d155-00d8-458e-afda-5be86f209550" containerName="manager" Sep 17 17:39:26.308600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.308573 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.341223 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.341191 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh"] Sep 17 17:39:26.341416 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.341390 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" containerName="manager" containerID="cri-o://9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941" gracePeriod=2 Sep 17 17:39:26.353178 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.352760 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67"] Sep 17 17:39:26.354783 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.354670 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh"] Sep 17 17:39:26.375654 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.375615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b448b3c-262d-47c9-b512-5c667926fff1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-flz67\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.375836 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.375702 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfns\" (UniqueName: \"kubernetes.io/projected/3b448b3c-262d-47c9-b512-5c667926fff1-kube-api-access-5bfns\") pod \"kuadrant-operator-controller-manager-76f8f686bf-flz67\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.382789 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.382759 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc"] Sep 17 17:39:26.383071 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.383056 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" containerName="manager" Sep 17 17:39:26.383071 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.383071 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" containerName="manager" Sep 17 17:39:26.383205 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.383126 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" containerName="manager" Sep 17 17:39:26.385932 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.385909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.391219 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.391188 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.393226 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.393205 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.399272 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.399243 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.402329 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.402305 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87"] Sep 17 17:39:26.405325 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.405311 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:26.414460 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.414362 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc"] Sep 17 17:39:26.425847 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.425821 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87"] Sep 17 17:39:26.442917 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.442888 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.445393 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.445367 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.466969 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.466936 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.476121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.476099 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7l2l\" (UniqueName: \"kubernetes.io/projected/ac835e77-2abe-4e4b-97f0-e2a1235fdbd4-kube-api-access-p7l2l\") pod \"limitador-operator-controller-manager-74495c69f8-j5g87\" (UID: \"ac835e77-2abe-4e4b-97f0-e2a1235fdbd4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:26.476231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.476130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d713027c-9f89-42b7-bfd5-c772c4c6bf86-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lwrpc\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.476231 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.476155 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfns\" (UniqueName: \"kubernetes.io/projected/3b448b3c-262d-47c9-b512-5c667926fff1-kube-api-access-5bfns\") pod \"kuadrant-operator-controller-manager-76f8f686bf-flz67\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.476324 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.476226 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdrd\" (UniqueName: \"kubernetes.io/projected/d713027c-9f89-42b7-bfd5-c772c4c6bf86-kube-api-access-wcdrd\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lwrpc\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.476324 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.476298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b448b3c-262d-47c9-b512-5c667926fff1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-flz67\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.476579 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.476564 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b448b3c-262d-47c9-b512-5c667926fff1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-flz67\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.486728 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.486694 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.489404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.489378 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfns\" (UniqueName: \"kubernetes.io/projected/3b448b3c-262d-47c9-b512-5c667926fff1-kube-api-access-5bfns\") pod \"kuadrant-operator-controller-manager-76f8f686bf-flz67\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.490204 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.490178 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.504529 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.504495 2566 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 17 17:39:26.504529 ip-10-0-142-117 kubenswrapper[2566]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-4gxhm_kuadrant-system_01db3e4b-03b1-413f-ad17-932d2cd5a6ce_0(141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-4gxhm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0" Netns:"/var/run/netns/089befca-a995-4ed2-ae50-6cc0da996041" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-4gxhm;K8S_POD_INFRA_CONTAINER_ID=141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0;K8S_POD_UID=01db3e4b-03b1-413f-ad17-932d2cd5a6ce" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm/01db3e4b-03b1-413f-ad17-932d2cd5a6ce:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm 141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:18 [10.132.0.24/23] Sep 17 17:39:26.504529 ip-10-0-142-117 kubenswrapper[2566]: ' Sep 17 17:39:26.504529 ip-10-0-142-117 kubenswrapper[2566]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 17 17:39:26.504529 ip-10-0-142-117 kubenswrapper[2566]: > Sep 17 17:39:26.504741 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.504545 2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 17 17:39:26.504741 ip-10-0-142-117 kubenswrapper[2566]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-4gxhm_kuadrant-system_01db3e4b-03b1-413f-ad17-932d2cd5a6ce_0(141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-4gxhm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0" Netns:"/var/run/netns/089befca-a995-4ed2-ae50-6cc0da996041" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-4gxhm;K8S_POD_INFRA_CONTAINER_ID=141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0;K8S_POD_UID=01db3e4b-03b1-413f-ad17-932d2cd5a6ce" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm/01db3e4b-03b1-413f-ad17-932d2cd5a6ce:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm 141dc44d85ae4ad56923516c337a126f00b354e95fa86fa0fe62db10cd1138d0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:18 [10.132.0.24/23] Sep 17 17:39:26.504741 ip-10-0-142-117 kubenswrapper[2566]: ' Sep 17 17:39:26.504741 ip-10-0-142-117 kubenswrapper[2566]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 17 17:39:26.504741 ip-10-0-142-117 kubenswrapper[2566]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.577489 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.577468 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kjqn\" (UniqueName: \"kubernetes.io/projected/0717d155-00d8-458e-afda-5be86f209550-kube-api-access-4kjqn\") pod \"0717d155-00d8-458e-afda-5be86f209550\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " Sep 17 17:39:26.577600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.577518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0717d155-00d8-458e-afda-5be86f209550-extensions-socket-volume\") pod \"0717d155-00d8-458e-afda-5be86f209550\" (UID: \"0717d155-00d8-458e-afda-5be86f209550\") " Sep 17 17:39:26.577600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.577562 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.577600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.577587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d713027c-9f89-42b7-bfd5-c772c4c6bf86-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lwrpc\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.577754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.577628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdrd\" (UniqueName: \"kubernetes.io/projected/d713027c-9f89-42b7-bfd5-c772c4c6bf86-kube-api-access-wcdrd\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lwrpc\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.577754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.577706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7l2l\" (UniqueName: \"kubernetes.io/projected/ac835e77-2abe-4e4b-97f0-e2a1235fdbd4-kube-api-access-p7l2l\") pod \"limitador-operator-controller-manager-74495c69f8-j5g87\" (UID: \"ac835e77-2abe-4e4b-97f0-e2a1235fdbd4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:26.578034 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.578012 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d713027c-9f89-42b7-bfd5-c772c4c6bf86-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lwrpc\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.578135 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.578105 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0717d155-00d8-458e-afda-5be86f209550-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0717d155-00d8-458e-afda-5be86f209550" (UID: "0717d155-00d8-458e-afda-5be86f209550"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:39:26.579505 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.579479 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0717d155-00d8-458e-afda-5be86f209550-kube-api-access-4kjqn" (OuterVolumeSpecName: "kube-api-access-4kjqn") pod "0717d155-00d8-458e-afda-5be86f209550" (UID: "0717d155-00d8-458e-afda-5be86f209550"). InnerVolumeSpecName "kube-api-access-4kjqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:39:26.580159 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.580137 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.582300 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.582278 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.603264 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.603212 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdrd\" (UniqueName: \"kubernetes.io/projected/d713027c-9f89-42b7-bfd5-c772c4c6bf86-kube-api-access-wcdrd\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lwrpc\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.603523 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.603504 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7l2l\" (UniqueName: \"kubernetes.io/projected/ac835e77-2abe-4e4b-97f0-e2a1235fdbd4-kube-api-access-p7l2l\") pod \"limitador-operator-controller-manager-74495c69f8-j5g87\" (UID: \"ac835e77-2abe-4e4b-97f0-e2a1235fdbd4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:26.619387 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.619360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:26.678407 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.678383 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gghbd\" (UniqueName: \"kubernetes.io/projected/2d15bb72-01cf-48db-b882-7dd9a5e31dba-kube-api-access-gghbd\") pod \"2d15bb72-01cf-48db-b882-7dd9a5e31dba\" (UID: \"2d15bb72-01cf-48db-b882-7dd9a5e31dba\") " Sep 17 17:39:26.678517 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.678503 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0717d155-00d8-458e-afda-5be86f209550-extensions-socket-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:26.678575 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.678517 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjqn\" (UniqueName: \"kubernetes.io/projected/0717d155-00d8-458e-afda-5be86f209550-kube-api-access-4kjqn\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:26.680376 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.680352 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d15bb72-01cf-48db-b882-7dd9a5e31dba-kube-api-access-gghbd" (OuterVolumeSpecName: "kube-api-access-gghbd") pod "2d15bb72-01cf-48db-b882-7dd9a5e31dba" (UID: "2d15bb72-01cf-48db-b882-7dd9a5e31dba"). InnerVolumeSpecName "kube-api-access-gghbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:39:26.733553 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.733524 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:26.740347 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.740332 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:26.779020 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.778991 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gghbd\" (UniqueName: \"kubernetes.io/projected/2d15bb72-01cf-48db-b882-7dd9a5e31dba-kube-api-access-gghbd\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:26.898933 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.898853 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67"] Sep 17 17:39:26.901880 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:26.901855 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b448b3c_262d_47c9_b512_5c667926fff1.slice/crio-87ecdcb4aa04df95517659e2b4684fcd035ccff56e6135d89c9e390ba7791a92 WatchSource:0}: Error finding container 87ecdcb4aa04df95517659e2b4684fcd035ccff56e6135d89c9e390ba7791a92: Status 404 returned error can't find the container with id 87ecdcb4aa04df95517659e2b4684fcd035ccff56e6135d89c9e390ba7791a92 Sep 17 17:39:26.959099 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.959070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" event={"ID":"3b448b3c-262d-47c9-b512-5c667926fff1","Type":"ContainerStarted","Data":"87ecdcb4aa04df95517659e2b4684fcd035ccff56e6135d89c9e390ba7791a92"} Sep 17 17:39:26.960426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.960399 2566 generic.go:356] "Generic (PLEG): container finished" podID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" containerID="9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941" exitCode=0 Sep 17 17:39:26.960533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.960443 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.960533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.960488 2566 scope.go:117] "RemoveContainer" containerID="9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941" Sep 17 17:39:26.962056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.962032 2566 generic.go:356] "Generic (PLEG): container finished" podID="0717d155-00d8-458e-afda-5be86f209550" containerID="e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452" exitCode=0 Sep 17 17:39:26.962155 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.962083 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.962155 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.962100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.962888 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.962863 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.965594 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.965568 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.967716 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.967687 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.969381 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.969361 2566 scope.go:117] "RemoveContainer" containerID="9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941" Sep 17 17:39:26.969477 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.969368 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.969893 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.969670 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941\": container with ID starting with 9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941 not found: ID does not exist" containerID="9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941" Sep 17 17:39:26.969893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.969700 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941"} err="failed to get container status \"9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941\": rpc error: code = NotFound desc = could not find container \"9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941\": container with ID starting with 9b364cc3a48b0815c92889b44fbebfa377d9e667144420191012d916f113d941 not found: ID does not exist" Sep 17 17:39:26.969893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.969723 2566 scope.go:117] "RemoveContainer" containerID="e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452" Sep 17 17:39:26.970323 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.970299 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="01db3e4b-03b1-413f-ad17-932d2cd5a6ce" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.972285 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.972259 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.974610 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.974582 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="01db3e4b-03b1-413f-ad17-932d2cd5a6ce" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.976416 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.976389 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.977786 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.977771 2566 scope.go:117] "RemoveContainer" containerID="e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452" Sep 17 17:39:26.978076 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.978056 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452\": container with ID starting with e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452 not found: ID does not exist" containerID="e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452" Sep 17 17:39:26.978152 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:26.978081 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452"} err="failed to get container status \"e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452\": rpc error: code = NotFound desc = could not find container \"e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452\": container with ID starting with e258425e710ab125dd6065671e3947dc3ca3c6af46793fd639c72e0d32c73452 not found: ID does not exist" Sep 17 17:39:26.979230 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.979212 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:26.981036 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.981016 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="01db3e4b-03b1-413f-ad17-932d2cd5a6ce" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:26.983171 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.983150 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:26.984867 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:26.984850 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:27.019858 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.019837 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc"] Sep 17 17:39:27.022205 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:27.022152 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd713027c_9f89_42b7_bfd5_c772c4c6bf86.slice/crio-0fa396e2829c308cb864db7a50fab309c56ec59a401258412aecbce6bd1885f4 WatchSource:0}: Error finding container 0fa396e2829c308cb864db7a50fab309c56ec59a401258412aecbce6bd1885f4: Status 404 returned error can't find the container with id 0fa396e2829c308cb864db7a50fab309c56ec59a401258412aecbce6bd1885f4 Sep 17 17:39:27.023790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.023770 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87"] Sep 17 17:39:27.027069 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:27.027038 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac835e77_2abe_4e4b_97f0_e2a1235fdbd4.slice/crio-aa03818ff38a29645b716dc95e9760cb92b1ae2276bfaaa3acdc96e5149f2e35 WatchSource:0}: Error finding container aa03818ff38a29645b716dc95e9760cb92b1ae2276bfaaa3acdc96e5149f2e35: Status 404 returned error can't find the container with id aa03818ff38a29645b716dc95e9760cb92b1ae2276bfaaa3acdc96e5149f2e35 Sep 17 17:39:27.081003 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.080981 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-extensions-socket-volume\") pod \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " Sep 17 17:39:27.081124 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.081017 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f89q9\" (UniqueName: \"kubernetes.io/projected/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-kube-api-access-f89q9\") pod \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\" (UID: \"01db3e4b-03b1-413f-ad17-932d2cd5a6ce\") " Sep 17 17:39:27.081285 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.081262 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "01db3e4b-03b1-413f-ad17-932d2cd5a6ce" (UID: "01db3e4b-03b1-413f-ad17-932d2cd5a6ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:39:27.083260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.083217 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-kube-api-access-f89q9" (OuterVolumeSpecName: "kube-api-access-f89q9") pod "01db3e4b-03b1-413f-ad17-932d2cd5a6ce" (UID: "01db3e4b-03b1-413f-ad17-932d2cd5a6ce"). InnerVolumeSpecName "kube-api-access-f89q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:39:27.182239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.182167 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-extensions-socket-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:27.182239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.182193 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89q9\" (UniqueName: \"kubernetes.io/projected/01db3e4b-03b1-413f-ad17-932d2cd5a6ce-kube-api-access-f89q9\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:27.971059 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.971014 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" event={"ID":"3b448b3c-262d-47c9-b512-5c667926fff1","Type":"ContainerStarted","Data":"a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235"} Sep 17 17:39:27.971245 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.971157 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:27.973439 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:27.973413 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="01db3e4b-03b1-413f-ad17-932d2cd5a6ce" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:27.974872 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.974850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" event={"ID":"ac835e77-2abe-4e4b-97f0-e2a1235fdbd4","Type":"ContainerStarted","Data":"05e2e407b842ce8b9008e5d40b3cd230c09be1b43e11e8a51eca1c3cdcef7bc3"} Sep 17 17:39:27.975274 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.974891 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" event={"ID":"ac835e77-2abe-4e4b-97f0-e2a1235fdbd4","Type":"ContainerStarted","Data":"aa03818ff38a29645b716dc95e9760cb92b1ae2276bfaaa3acdc96e5149f2e35"} Sep 17 17:39:27.975274 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.974920 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:27.976366 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.976349 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:27.976468 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.976346 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" event={"ID":"d713027c-9f89-42b7-bfd5-c772c4c6bf86","Type":"ContainerStarted","Data":"d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f"} Sep 17 17:39:27.976468 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:27.976451 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" event={"ID":"d713027c-9f89-42b7-bfd5-c772c4c6bf86","Type":"ContainerStarted","Data":"0fa396e2829c308cb864db7a50fab309c56ec59a401258412aecbce6bd1885f4"} Sep 17 17:39:28.006060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.006015 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" podStartSLOduration=2.005999745 podStartE2EDuration="2.005999745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:28.005334244 +0000 UTC m=+504.567499103" watchObservedRunningTime="2026-09-17 17:39:28.005999745 +0000 UTC m=+504.568164623" Sep 17 17:39:28.007330 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:28.007304 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:28.009050 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:28.009028 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:28.037188 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.037148 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" podStartSLOduration=2.037131595 podStartE2EDuration="2.037131595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:28.036281379 +0000 UTC m=+504.598446238" watchObservedRunningTime="2026-09-17 17:39:28.037131595 +0000 UTC m=+504.599296456" Sep 17 17:39:28.039251 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:28.039230 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4gxhm\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="01db3e4b-03b1-413f-ad17-932d2cd5a6ce" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4gxhm" Sep 17 17:39:28.041070 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:28.041053 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mvqmb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="0717d155-00d8-458e-afda-5be86f209550" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mvqmb" Sep 17 17:39:28.042935 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:28.042915 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvxrh\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvxrh" Sep 17 17:39:28.075251 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.075203 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" podStartSLOduration=2.075189051 podStartE2EDuration="2.075189051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:28.074374451 +0000 UTC m=+504.636539334" watchObservedRunningTime="2026-09-17 17:39:28.075189051 +0000 UTC m=+504.637353936" Sep 17 17:39:28.149796 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.149773 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01db3e4b-03b1-413f-ad17-932d2cd5a6ce" path="/var/lib/kubelet/pods/01db3e4b-03b1-413f-ad17-932d2cd5a6ce/volumes" Sep 17 17:39:28.150034 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.150021 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0717d155-00d8-458e-afda-5be86f209550" path="/var/lib/kubelet/pods/0717d155-00d8-458e-afda-5be86f209550/volumes" Sep 17 17:39:28.150296 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.150285 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d15bb72-01cf-48db-b882-7dd9a5e31dba" path="/var/lib/kubelet/pods/2d15bb72-01cf-48db-b882-7dd9a5e31dba/volumes" Sep 17 17:39:28.980496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:28.980459 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:38.982845 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:38.982817 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:38.983380 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:38.983098 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5g87" Sep 17 17:39:39.988078 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:39.988052 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:40.154189 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.154162 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67"] Sep 17 17:39:40.154356 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.154312 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" podUID="3b448b3c-262d-47c9-b512-5c667926fff1" containerName="manager" containerID="cri-o://a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235" gracePeriod=10 Sep 17 17:39:40.390554 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.390530 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:40.486216 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.486185 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b448b3c-262d-47c9-b512-5c667926fff1-extensions-socket-volume\") pod \"3b448b3c-262d-47c9-b512-5c667926fff1\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " Sep 17 17:39:40.486354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.486248 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bfns\" (UniqueName: \"kubernetes.io/projected/3b448b3c-262d-47c9-b512-5c667926fff1-kube-api-access-5bfns\") pod \"3b448b3c-262d-47c9-b512-5c667926fff1\" (UID: \"3b448b3c-262d-47c9-b512-5c667926fff1\") " Sep 17 17:39:40.486587 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.486565 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b448b3c-262d-47c9-b512-5c667926fff1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3b448b3c-262d-47c9-b512-5c667926fff1" (UID: "3b448b3c-262d-47c9-b512-5c667926fff1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:39:40.488220 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.488188 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b448b3c-262d-47c9-b512-5c667926fff1-kube-api-access-5bfns" (OuterVolumeSpecName: "kube-api-access-5bfns") pod "3b448b3c-262d-47c9-b512-5c667926fff1" (UID: "3b448b3c-262d-47c9-b512-5c667926fff1"). InnerVolumeSpecName "kube-api-access-5bfns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:39:40.587543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.587519 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b448b3c-262d-47c9-b512-5c667926fff1-extensions-socket-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:40.587543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.587539 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfns\" (UniqueName: \"kubernetes.io/projected/3b448b3c-262d-47c9-b512-5c667926fff1-kube-api-access-5bfns\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:40.753868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.753837 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69"] Sep 17 17:39:40.754176 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.754163 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b448b3c-262d-47c9-b512-5c667926fff1" containerName="manager" Sep 17 17:39:40.754220 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.754177 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b448b3c-262d-47c9-b512-5c667926fff1" containerName="manager" Sep 17 17:39:40.754257 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.754228 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b448b3c-262d-47c9-b512-5c667926fff1" containerName="manager" Sep 17 17:39:40.755900 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.755885 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:40.776868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.776845 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69"] Sep 17 17:39:40.889926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.889858 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5tw69\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:40.889926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.889894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gdf\" (UniqueName: \"kubernetes.io/projected/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-kube-api-access-59gdf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5tw69\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:40.991144 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.991107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5tw69\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:40.991144 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.991154 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gdf\" (UniqueName: \"kubernetes.io/projected/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-kube-api-access-59gdf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5tw69\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:40.991489 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:40.991449 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5tw69\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:41.005332 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.005313 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gdf\" (UniqueName: \"kubernetes.io/projected/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-kube-api-access-59gdf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5tw69\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:41.033272 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.033241 2566 generic.go:356] "Generic (PLEG): container finished" podID="3b448b3c-262d-47c9-b512-5c667926fff1" containerID="a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235" exitCode=0 Sep 17 17:39:41.033370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.033285 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" Sep 17 17:39:41.033370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.033293 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" event={"ID":"3b448b3c-262d-47c9-b512-5c667926fff1","Type":"ContainerDied","Data":"a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235"} Sep 17 17:39:41.033370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.033337 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67" event={"ID":"3b448b3c-262d-47c9-b512-5c667926fff1","Type":"ContainerDied","Data":"87ecdcb4aa04df95517659e2b4684fcd035ccff56e6135d89c9e390ba7791a92"} Sep 17 17:39:41.033370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.033363 2566 scope.go:117] "RemoveContainer" containerID="a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235" Sep 17 17:39:41.041191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.041176 2566 scope.go:117] "RemoveContainer" containerID="a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235" Sep 17 17:39:41.041446 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:41.041429 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235\": container with ID starting with a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235 not found: ID does not exist" containerID="a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235" Sep 17 17:39:41.041531 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.041452 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235"} err="failed to get container status \"a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235\": rpc error: code = NotFound desc = could not find container \"a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235\": container with ID starting with a56c9625d604874564c237a76ca9b9e118725fbfae91eafd3c79e3cc6de51235 not found: ID does not exist" Sep 17 17:39:41.061233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.061214 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67"] Sep 17 17:39:41.065360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.065342 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:41.066846 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.066829 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-flz67"] Sep 17 17:39:41.344656 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:41.344585 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69"] Sep 17 17:39:41.347410 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:39:41.347378 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daa7083_d1ec_4069_a7d6_9a3fec0574e6.slice/crio-813b40557ab6997dfe2541570b182ef1ae70c7a9500914f0fcbf4e9da5105b0f WatchSource:0}: Error finding container 813b40557ab6997dfe2541570b182ef1ae70c7a9500914f0fcbf4e9da5105b0f: Status 404 returned error can't find the container with id 813b40557ab6997dfe2541570b182ef1ae70c7a9500914f0fcbf4e9da5105b0f Sep 17 17:39:42.039969 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:42.039911 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" event={"ID":"5daa7083-d1ec-4069-a7d6-9a3fec0574e6","Type":"ContainerStarted","Data":"a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e"} Sep 17 17:39:42.040382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:42.039995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" event={"ID":"5daa7083-d1ec-4069-a7d6-9a3fec0574e6","Type":"ContainerStarted","Data":"813b40557ab6997dfe2541570b182ef1ae70c7a9500914f0fcbf4e9da5105b0f"} Sep 17 17:39:42.040382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:42.040029 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:42.062731 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:42.062682 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" podStartSLOduration=2.062665887 podStartE2EDuration="2.062665887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:42.06177371 +0000 UTC m=+518.623938570" watchObservedRunningTime="2026-09-17 17:39:42.062665887 +0000 UTC m=+518.624830748" Sep 17 17:39:42.149224 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:42.149196 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b448b3c-262d-47c9-b512-5c667926fff1" path="/var/lib/kubelet/pods/3b448b3c-262d-47c9-b512-5c667926fff1/volumes" Sep 17 17:39:53.047631 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.047595 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:39:53.121421 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.121389 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc"] Sep 17 17:39:53.121601 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.121576 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" podUID="d713027c-9f89-42b7-bfd5-c772c4c6bf86" containerName="manager" containerID="cri-o://d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f" gracePeriod=10 Sep 17 17:39:53.347928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.347903 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:53.490656 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.490622 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcdrd\" (UniqueName: \"kubernetes.io/projected/d713027c-9f89-42b7-bfd5-c772c4c6bf86-kube-api-access-wcdrd\") pod \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " Sep 17 17:39:53.490814 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.490674 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d713027c-9f89-42b7-bfd5-c772c4c6bf86-extensions-socket-volume\") pod \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\" (UID: \"d713027c-9f89-42b7-bfd5-c772c4c6bf86\") " Sep 17 17:39:53.491091 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.491059 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713027c-9f89-42b7-bfd5-c772c4c6bf86-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d713027c-9f89-42b7-bfd5-c772c4c6bf86" (UID: "d713027c-9f89-42b7-bfd5-c772c4c6bf86"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:39:53.492647 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.492623 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d713027c-9f89-42b7-bfd5-c772c4c6bf86-kube-api-access-wcdrd" (OuterVolumeSpecName: "kube-api-access-wcdrd") pod "d713027c-9f89-42b7-bfd5-c772c4c6bf86" (UID: "d713027c-9f89-42b7-bfd5-c772c4c6bf86"). InnerVolumeSpecName "kube-api-access-wcdrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:39:53.591141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.591084 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d713027c-9f89-42b7-bfd5-c772c4c6bf86-extensions-socket-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:53.591141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:53.591107 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcdrd\" (UniqueName: \"kubernetes.io/projected/d713027c-9f89-42b7-bfd5-c772c4c6bf86-kube-api-access-wcdrd\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:39:54.094199 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.094161 2566 generic.go:356] "Generic (PLEG): container finished" podID="d713027c-9f89-42b7-bfd5-c772c4c6bf86" containerID="d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f" exitCode=0 Sep 17 17:39:54.094655 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.094212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" event={"ID":"d713027c-9f89-42b7-bfd5-c772c4c6bf86","Type":"ContainerDied","Data":"d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f"} Sep 17 17:39:54.094655 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.094214 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" Sep 17 17:39:54.094655 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.094255 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc" event={"ID":"d713027c-9f89-42b7-bfd5-c772c4c6bf86","Type":"ContainerDied","Data":"0fa396e2829c308cb864db7a50fab309c56ec59a401258412aecbce6bd1885f4"} Sep 17 17:39:54.094655 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.094277 2566 scope.go:117] "RemoveContainer" containerID="d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f" Sep 17 17:39:54.104781 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.104760 2566 scope.go:117] "RemoveContainer" containerID="d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f" Sep 17 17:39:54.105079 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:39:54.105058 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f\": container with ID starting with d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f not found: ID does not exist" containerID="d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f" Sep 17 17:39:54.105135 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.105089 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f"} err="failed to get container status \"d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f\": rpc error: code = NotFound desc = could not find container \"d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f\": container with ID starting with d246fb9217b240efce7727cbcb01bc516e32df91e8bdbcf928259609b1e5d03f not found: ID does not exist" Sep 17 17:39:54.122321 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.122295 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc"] Sep 17 17:39:54.123496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.123477 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lwrpc"] Sep 17 17:39:54.149512 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:39:54.149485 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d713027c-9f89-42b7-bfd5-c772c4c6bf86" path="/var/lib/kubelet/pods/d713027c-9f89-42b7-bfd5-c772c4c6bf86/volumes" Sep 17 17:40:08.222337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.222304 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:08.222694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.222591 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d713027c-9f89-42b7-bfd5-c772c4c6bf86" containerName="manager" Sep 17 17:40:08.222694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.222604 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d713027c-9f89-42b7-bfd5-c772c4c6bf86" containerName="manager" Sep 17 17:40:08.222694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.222659 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d713027c-9f89-42b7-bfd5-c772c4c6bf86" containerName="manager" Sep 17 17:40:08.226948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.226929 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.229241 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.229222 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 17 17:40:08.229241 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.229231 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gqjh7\"" Sep 17 17:40:08.233976 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.233931 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:08.297236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.297201 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e162198-0cb2-42ad-b9e7-7f701857bdcc-config-file\") pod \"limitador-limitador-8588749977-wfthc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.297377 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.297302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpp8d\" (UniqueName: \"kubernetes.io/projected/0e162198-0cb2-42ad-b9e7-7f701857bdcc-kube-api-access-vpp8d\") pod \"limitador-limitador-8588749977-wfthc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.320706 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.320676 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:08.397908 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.397879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e162198-0cb2-42ad-b9e7-7f701857bdcc-config-file\") pod \"limitador-limitador-8588749977-wfthc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.398097 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.397928 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp8d\" (UniqueName: \"kubernetes.io/projected/0e162198-0cb2-42ad-b9e7-7f701857bdcc-kube-api-access-vpp8d\") pod \"limitador-limitador-8588749977-wfthc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.398553 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.398527 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e162198-0cb2-42ad-b9e7-7f701857bdcc-config-file\") pod \"limitador-limitador-8588749977-wfthc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.407604 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.407573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp8d\" (UniqueName: \"kubernetes.io/projected/0e162198-0cb2-42ad-b9e7-7f701857bdcc-kube-api-access-vpp8d\") pod \"limitador-limitador-8588749977-wfthc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.537302 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.537216 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:08.806763 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:08.806736 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:08.809709 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:40:08.809680 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e162198_0cb2_42ad_b9e7_7f701857bdcc.slice/crio-a5cd9ef63a98f34c981b54de30402986d056aac57a4a177afcafdc35d6917ae4 WatchSource:0}: Error finding container a5cd9ef63a98f34c981b54de30402986d056aac57a4a177afcafdc35d6917ae4: Status 404 returned error can't find the container with id a5cd9ef63a98f34c981b54de30402986d056aac57a4a177afcafdc35d6917ae4 Sep 17 17:40:09.155497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.155416 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" event={"ID":"0e162198-0cb2-42ad-b9e7-7f701857bdcc","Type":"ContainerStarted","Data":"a5cd9ef63a98f34c981b54de30402986d056aac57a4a177afcafdc35d6917ae4"} Sep 17 17:40:09.226211 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.226181 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cv8gl"] Sep 17 17:40:09.230610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.230591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:09.236053 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.236025 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zb79d\"" Sep 17 17:40:09.239614 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.239591 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cv8gl"] Sep 17 17:40:09.304411 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.304386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nfj\" (UniqueName: \"kubernetes.io/projected/051bdb77-47a0-406b-a7de-d98a6c79c346-kube-api-access-w6nfj\") pod \"authorino-59c974bd48-cv8gl\" (UID: \"051bdb77-47a0-406b-a7de-d98a6c79c346\") " pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:09.404743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.404716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nfj\" (UniqueName: \"kubernetes.io/projected/051bdb77-47a0-406b-a7de-d98a6c79c346-kube-api-access-w6nfj\") pod \"authorino-59c974bd48-cv8gl\" (UID: \"051bdb77-47a0-406b-a7de-d98a6c79c346\") " pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:09.413636 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.413581 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nfj\" (UniqueName: \"kubernetes.io/projected/051bdb77-47a0-406b-a7de-d98a6c79c346-kube-api-access-w6nfj\") pod \"authorino-59c974bd48-cv8gl\" (UID: \"051bdb77-47a0-406b-a7de-d98a6c79c346\") " pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:09.553552 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.553515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:09.829778 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:09.829748 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cv8gl"] Sep 17 17:40:09.894709 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:40:09.894675 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod051bdb77_47a0_406b_a7de_d98a6c79c346.slice/crio-63748cefdb8db4688e53129775b08e842da5554be6204ff832e439a2401249ea WatchSource:0}: Error finding container 63748cefdb8db4688e53129775b08e842da5554be6204ff832e439a2401249ea: Status 404 returned error can't find the container with id 63748cefdb8db4688e53129775b08e842da5554be6204ff832e439a2401249ea Sep 17 17:40:10.160146 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:10.160063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" event={"ID":"0e162198-0cb2-42ad-b9e7-7f701857bdcc","Type":"ContainerStarted","Data":"dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6"} Sep 17 17:40:10.160146 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:10.160128 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:10.161267 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:10.161245 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cv8gl" event={"ID":"051bdb77-47a0-406b-a7de-d98a6c79c346","Type":"ContainerStarted","Data":"63748cefdb8db4688e53129775b08e842da5554be6204ff832e439a2401249ea"} Sep 17 17:40:10.179374 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:10.179334 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" podStartSLOduration=1.042339437 podStartE2EDuration="2.179320926s" totalImagesPullingTime="1.136981489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:10.178865793 +0000 UTC m=+546.741030657" watchObservedRunningTime="2026-09-17 17:40:10.179320926 +0000 UTC m=+546.741485787" Sep 17 17:40:12.173775 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:12.173729 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cv8gl" event={"ID":"051bdb77-47a0-406b-a7de-d98a6c79c346","Type":"ContainerStarted","Data":"a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1"} Sep 17 17:40:12.192732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:12.192687 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cv8gl" podStartSLOduration=1.531980468 podStartE2EDuration="3.192673041s" totalImagesPullingTime="1.660692573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:12.191350185 +0000 UTC m=+548.753515070" watchObservedRunningTime="2026-09-17 17:40:12.192673041 +0000 UTC m=+548.754837902" Sep 17 17:40:21.167320 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:21.167288 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:23.252495 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:23.252455 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:23.252913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:23.252752 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" podUID="0e162198-0cb2-42ad-b9e7-7f701857bdcc" containerName="limitador" containerID="cri-o://dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6" gracePeriod=30 Sep 17 17:40:24.194294 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.194269 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:24.216128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.216101 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e162198-0cb2-42ad-b9e7-7f701857bdcc-config-file\") pod \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " Sep 17 17:40:24.216272 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.216167 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpp8d\" (UniqueName: \"kubernetes.io/projected/0e162198-0cb2-42ad-b9e7-7f701857bdcc-kube-api-access-vpp8d\") pod \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\" (UID: \"0e162198-0cb2-42ad-b9e7-7f701857bdcc\") " Sep 17 17:40:24.216470 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.216450 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e162198-0cb2-42ad-b9e7-7f701857bdcc-config-file" (OuterVolumeSpecName: "config-file") pod "0e162198-0cb2-42ad-b9e7-7f701857bdcc" (UID: "0e162198-0cb2-42ad-b9e7-7f701857bdcc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:40:24.218303 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.218269 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e162198-0cb2-42ad-b9e7-7f701857bdcc-kube-api-access-vpp8d" (OuterVolumeSpecName: "kube-api-access-vpp8d") pod "0e162198-0cb2-42ad-b9e7-7f701857bdcc" (UID: "0e162198-0cb2-42ad-b9e7-7f701857bdcc"). InnerVolumeSpecName "kube-api-access-vpp8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:40:24.223423 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.223393 2566 generic.go:356] "Generic (PLEG): container finished" podID="0e162198-0cb2-42ad-b9e7-7f701857bdcc" containerID="dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6" exitCode=0 Sep 17 17:40:24.223423 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.223442 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" Sep 17 17:40:24.223637 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.223512 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" event={"ID":"0e162198-0cb2-42ad-b9e7-7f701857bdcc","Type":"ContainerDied","Data":"dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6"} Sep 17 17:40:24.223637 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.223560 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfthc" event={"ID":"0e162198-0cb2-42ad-b9e7-7f701857bdcc","Type":"ContainerDied","Data":"a5cd9ef63a98f34c981b54de30402986d056aac57a4a177afcafdc35d6917ae4"} Sep 17 17:40:24.223637 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.223584 2566 scope.go:117] "RemoveContainer" containerID="dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6" Sep 17 17:40:24.231637 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.231621 2566 scope.go:117] "RemoveContainer" containerID="dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6" Sep 17 17:40:24.231926 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:24.231906 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6\": container with ID starting with dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6 not found: ID does not exist" containerID="dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6" Sep 17 17:40:24.232018 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.231948 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6"} err="failed to get container status \"dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6\": rpc error: code = NotFound desc = could not find container \"dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6\": container with ID starting with dcd5c3853e987f86dc996bfc08f4dfabce157aebbfa85eb24ca50609fea0d8b6 not found: ID does not exist" Sep 17 17:40:24.244275 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.244254 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:24.250545 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.250524 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfthc"] Sep 17 17:40:24.317137 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.317067 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpp8d\" (UniqueName: \"kubernetes.io/projected/0e162198-0cb2-42ad-b9e7-7f701857bdcc-kube-api-access-vpp8d\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:40:24.317137 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:24.317092 2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e162198-0cb2-42ad-b9e7-7f701857bdcc-config-file\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:40:26.149331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.149290 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e162198-0cb2-42ad-b9e7-7f701857bdcc" path="/var/lib/kubelet/pods/0e162198-0cb2-42ad-b9e7-7f701857bdcc/volumes" Sep 17 17:40:26.311239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.311204 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rrk87"] Sep 17 17:40:26.311568 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.311551 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e162198-0cb2-42ad-b9e7-7f701857bdcc" containerName="limitador" Sep 17 17:40:26.311662 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.311571 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e162198-0cb2-42ad-b9e7-7f701857bdcc" containerName="limitador" Sep 17 17:40:26.311662 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.311646 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e162198-0cb2-42ad-b9e7-7f701857bdcc" containerName="limitador" Sep 17 17:40:26.315832 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.315812 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.318714 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.318695 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 17 17:40:26.318946 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.318933 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 17:40:26.319353 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.319335 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 17:40:26.319611 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.319597 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ndgsw\"" Sep 17 17:40:26.324539 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.324519 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rrk87"] Sep 17 17:40:26.432482 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.432409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1db38631-3f8d-4008-afda-3ce44c78112d-data\") pod \"postgres-69fb4594cf-rrk87\" (UID: \"1db38631-3f8d-4008-afda-3ce44c78112d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.432482 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.432470 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7h6\" (UniqueName: \"kubernetes.io/projected/1db38631-3f8d-4008-afda-3ce44c78112d-kube-api-access-sj7h6\") pod \"postgres-69fb4594cf-rrk87\" (UID: \"1db38631-3f8d-4008-afda-3ce44c78112d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.533647 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.533614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7h6\" (UniqueName: \"kubernetes.io/projected/1db38631-3f8d-4008-afda-3ce44c78112d-kube-api-access-sj7h6\") pod \"postgres-69fb4594cf-rrk87\" (UID: \"1db38631-3f8d-4008-afda-3ce44c78112d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.533772 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.533663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1db38631-3f8d-4008-afda-3ce44c78112d-data\") pod \"postgres-69fb4594cf-rrk87\" (UID: \"1db38631-3f8d-4008-afda-3ce44c78112d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.534061 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.534045 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1db38631-3f8d-4008-afda-3ce44c78112d-data\") pod \"postgres-69fb4594cf-rrk87\" (UID: \"1db38631-3f8d-4008-afda-3ce44c78112d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.543180 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.543158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7h6\" (UniqueName: \"kubernetes.io/projected/1db38631-3f8d-4008-afda-3ce44c78112d-kube-api-access-sj7h6\") pod \"postgres-69fb4594cf-rrk87\" (UID: \"1db38631-3f8d-4008-afda-3ce44c78112d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.625682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.625658 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:26.894625 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:26.894536 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rrk87"] Sep 17 17:40:26.897343 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:40:26.897314 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db38631_3f8d_4008_afda_3ce44c78112d.slice/crio-06ce2de68828d15840498b87c05782a6f0b29efa95661eeff763b035834472e0 WatchSource:0}: Error finding container 06ce2de68828d15840498b87c05782a6f0b29efa95661eeff763b035834472e0: Status 404 returned error can't find the container with id 06ce2de68828d15840498b87c05782a6f0b29efa95661eeff763b035834472e0 Sep 17 17:40:27.237536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:27.237443 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" event={"ID":"1db38631-3f8d-4008-afda-3ce44c78112d","Type":"ContainerStarted","Data":"06ce2de68828d15840498b87c05782a6f0b29efa95661eeff763b035834472e0"} Sep 17 17:40:32.262965 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:32.262928 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" event={"ID":"1db38631-3f8d-4008-afda-3ce44c78112d","Type":"ContainerStarted","Data":"252892f3b6c0d289ca4ffcb22f1860e10c97c750c5bb5c097d20a4ae8c02f8c9"} Sep 17 17:40:32.263340 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:32.263085 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:32.285743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:32.285694 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" podStartSLOduration=1.571891907 podStartE2EDuration="6.285678806s" totalImagesPullingTime="4.713786899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:32.283614441 +0000 UTC m=+568.845779301" watchObservedRunningTime="2026-09-17 17:40:32.285678806 +0000 UTC m=+568.847843666" Sep 17 17:40:38.295976 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:38.295926 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rrk87" Sep 17 17:40:39.076824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.076792 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bzncl"] Sep 17 17:40:39.082365 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.082341 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.084939 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.084917 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 17 17:40:39.089419 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.089391 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bzncl"] Sep 17 17:40:39.139405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.139381 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7456f34-70c7-4705-adb6-e5222488e0fa-tls-cert\") pod \"authorino-6d4dfc649f-bzncl\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.139530 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.139414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnt9\" (UniqueName: \"kubernetes.io/projected/d7456f34-70c7-4705-adb6-e5222488e0fa-kube-api-access-6rnt9\") pod \"authorino-6d4dfc649f-bzncl\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.205685 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.205655 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bzncl"] Sep 17 17:40:39.205875 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:39.205857 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6rnt9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-bzncl" podUID="d7456f34-70c7-4705-adb6-e5222488e0fa" Sep 17 17:40:39.236863 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.236832 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55674c7f-psqxl"] Sep 17 17:40:39.240693 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.240674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.240824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.240663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7456f34-70c7-4705-adb6-e5222488e0fa-tls-cert\") pod \"authorino-6d4dfc649f-bzncl\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.240824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.240791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnt9\" (UniqueName: \"kubernetes.io/projected/d7456f34-70c7-4705-adb6-e5222488e0fa-kube-api-access-6rnt9\") pod \"authorino-6d4dfc649f-bzncl\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.243306 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.243281 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7456f34-70c7-4705-adb6-e5222488e0fa-tls-cert\") pod \"authorino-6d4dfc649f-bzncl\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.247100 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.247068 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55674c7f-psqxl"] Sep 17 17:40:39.250436 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.250418 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnt9\" (UniqueName: \"kubernetes.io/projected/d7456f34-70c7-4705-adb6-e5222488e0fa-kube-api-access-6rnt9\") pod \"authorino-6d4dfc649f-bzncl\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.292497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.292465 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.302566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.302541 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:39.342271 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.342183 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pp2\" (UniqueName: \"kubernetes.io/projected/69665146-80d3-49cf-b91c-eaf8149581e3-kube-api-access-p7pp2\") pod \"authorino-55674c7f-psqxl\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.342271 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.342238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69665146-80d3-49cf-b91c-eaf8149581e3-tls-cert\") pod \"authorino-55674c7f-psqxl\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.443203 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.443171 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7456f34-70c7-4705-adb6-e5222488e0fa-tls-cert\") pod \"d7456f34-70c7-4705-adb6-e5222488e0fa\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " Sep 17 17:40:39.443203 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.443206 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rnt9\" (UniqueName: \"kubernetes.io/projected/d7456f34-70c7-4705-adb6-e5222488e0fa-kube-api-access-6rnt9\") pod \"d7456f34-70c7-4705-adb6-e5222488e0fa\" (UID: \"d7456f34-70c7-4705-adb6-e5222488e0fa\") " Sep 17 17:40:39.443410 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.443302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69665146-80d3-49cf-b91c-eaf8149581e3-tls-cert\") pod \"authorino-55674c7f-psqxl\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.443410 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.443349 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pp2\" (UniqueName: \"kubernetes.io/projected/69665146-80d3-49cf-b91c-eaf8149581e3-kube-api-access-p7pp2\") pod \"authorino-55674c7f-psqxl\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.445330 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.445293 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7456f34-70c7-4705-adb6-e5222488e0fa-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d7456f34-70c7-4705-adb6-e5222488e0fa" (UID: "d7456f34-70c7-4705-adb6-e5222488e0fa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:40:39.445330 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.445293 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7456f34-70c7-4705-adb6-e5222488e0fa-kube-api-access-6rnt9" (OuterVolumeSpecName: "kube-api-access-6rnt9") pod "d7456f34-70c7-4705-adb6-e5222488e0fa" (UID: "d7456f34-70c7-4705-adb6-e5222488e0fa"). InnerVolumeSpecName "kube-api-access-6rnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:40:39.445676 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.445654 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69665146-80d3-49cf-b91c-eaf8149581e3-tls-cert\") pod \"authorino-55674c7f-psqxl\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.452129 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.452108 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pp2\" (UniqueName: \"kubernetes.io/projected/69665146-80d3-49cf-b91c-eaf8149581e3-kube-api-access-p7pp2\") pod \"authorino-55674c7f-psqxl\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.544206 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.544181 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnt9\" (UniqueName: \"kubernetes.io/projected/d7456f34-70c7-4705-adb6-e5222488e0fa-kube-api-access-6rnt9\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:40:39.544206 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.544203 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7456f34-70c7-4705-adb6-e5222488e0fa-tls-cert\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:40:39.563128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.563107 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:40:39.831107 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:39.831076 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55674c7f-psqxl"] Sep 17 17:40:39.833999 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:40:39.833970 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69665146_80d3_49cf_b91c_eaf8149581e3.slice/crio-5a0cb566bedabb56201261ed5030e822ad14988f75a563610e7c2f865a36aa4c WatchSource:0}: Error finding container 5a0cb566bedabb56201261ed5030e822ad14988f75a563610e7c2f865a36aa4c: Status 404 returned error can't find the container with id 5a0cb566bedabb56201261ed5030e822ad14988f75a563610e7c2f865a36aa4c Sep 17 17:40:40.298560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:40.298534 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bzncl" Sep 17 17:40:40.298560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:40.298548 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55674c7f-psqxl" event={"ID":"69665146-80d3-49cf-b91c-eaf8149581e3","Type":"ContainerStarted","Data":"5a0cb566bedabb56201261ed5030e822ad14988f75a563610e7c2f865a36aa4c"} Sep 17 17:40:40.328279 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:40.328253 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bzncl"] Sep 17 17:40:40.332582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:40.332557 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bzncl"] Sep 17 17:40:41.304680 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.304643 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55674c7f-psqxl" event={"ID":"69665146-80d3-49cf-b91c-eaf8149581e3","Type":"ContainerStarted","Data":"f9243440ee21c81562f8f3d2445e9fec00446fdf0be776ea9b9bc32651a17b52"} Sep 17 17:40:41.332408 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.332361 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55674c7f-psqxl" podStartSLOduration=1.780041091 podStartE2EDuration="2.332345313s" totalImagesPullingTime="552.304222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:41.330178627 +0000 UTC m=+577.892343487" watchObservedRunningTime="2026-09-17 17:40:41.332345313 +0000 UTC m=+577.894510170" Sep 17 17:40:41.416857 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.416823 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cv8gl"] Sep 17 17:40:41.417021 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.417001 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cv8gl" podUID="051bdb77-47a0-406b-a7de-d98a6c79c346" containerName="authorino" containerID="cri-o://a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1" gracePeriod=30 Sep 17 17:40:41.661692 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.661671 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:41.762236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.762204 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6nfj\" (UniqueName: \"kubernetes.io/projected/051bdb77-47a0-406b-a7de-d98a6c79c346-kube-api-access-w6nfj\") pod \"051bdb77-47a0-406b-a7de-d98a6c79c346\" (UID: \"051bdb77-47a0-406b-a7de-d98a6c79c346\") " Sep 17 17:40:41.764238 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.764208 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051bdb77-47a0-406b-a7de-d98a6c79c346-kube-api-access-w6nfj" (OuterVolumeSpecName: "kube-api-access-w6nfj") pod "051bdb77-47a0-406b-a7de-d98a6c79c346" (UID: "051bdb77-47a0-406b-a7de-d98a6c79c346"). InnerVolumeSpecName "kube-api-access-w6nfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:40:41.863550 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:41.863483 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6nfj\" (UniqueName: \"kubernetes.io/projected/051bdb77-47a0-406b-a7de-d98a6c79c346-kube-api-access-w6nfj\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:40:42.149592 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.149520 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7456f34-70c7-4705-adb6-e5222488e0fa" path="/var/lib/kubelet/pods/d7456f34-70c7-4705-adb6-e5222488e0fa/volumes" Sep 17 17:40:42.309966 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.309932 2566 generic.go:356] "Generic (PLEG): container finished" podID="051bdb77-47a0-406b-a7de-d98a6c79c346" containerID="a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1" exitCode=0 Sep 17 17:40:42.310111 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.309988 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cv8gl" Sep 17 17:40:42.310111 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.310018 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cv8gl" event={"ID":"051bdb77-47a0-406b-a7de-d98a6c79c346","Type":"ContainerDied","Data":"a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1"} Sep 17 17:40:42.310111 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.310066 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cv8gl" event={"ID":"051bdb77-47a0-406b-a7de-d98a6c79c346","Type":"ContainerDied","Data":"63748cefdb8db4688e53129775b08e842da5554be6204ff832e439a2401249ea"} Sep 17 17:40:42.310111 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.310086 2566 scope.go:117] "RemoveContainer" containerID="a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1" Sep 17 17:40:42.318421 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.318404 2566 scope.go:117] "RemoveContainer" containerID="a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1" Sep 17 17:40:42.318690 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:42.318675 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1\": container with ID starting with a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1 not found: ID does not exist" containerID="a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1" Sep 17 17:40:42.318731 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.318698 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1"} err="failed to get container status \"a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1\": rpc error: code = NotFound desc = could not find container \"a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1\": container with ID starting with a795eed73088294305ba9d7f6e396e428006669044cf21dff59390608b890bb1 not found: ID does not exist" Sep 17 17:40:42.326832 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.326812 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cv8gl"] Sep 17 17:40:42.330972 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:42.330923 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cv8gl"] Sep 17 17:40:44.149046 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:44.149010 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051bdb77-47a0-406b-a7de-d98a6c79c346" path="/var/lib/kubelet/pods/051bdb77-47a0-406b-a7de-d98a6c79c346/volumes" Sep 17 17:40:46.813188 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.813155 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74ccbb7bf7-btjnv"] Sep 17 17:40:46.813603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.813523 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051bdb77-47a0-406b-a7de-d98a6c79c346" containerName="authorino" Sep 17 17:40:46.813603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.813536 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="051bdb77-47a0-406b-a7de-d98a6c79c346" containerName="authorino" Sep 17 17:40:46.813603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.813590 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="051bdb77-47a0-406b-a7de-d98a6c79c346" containerName="authorino" Sep 17 17:40:46.832340 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.832307 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74ccbb7bf7-btjnv"] Sep 17 17:40:46.832482 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.832394 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:46.835130 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.835103 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 17:40:46.836763 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.836734 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 17:40:46.836893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.836873 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tcnq8\"" Sep 17 17:40:46.837051 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.837031 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 17:40:46.903183 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.903156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:46.903362 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.903205 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-metrics-certs\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:46.903362 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:46.903233 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4r9\" (UniqueName: \"kubernetes.io/projected/36db6e1f-6ebc-444a-89bf-4db7b32457ca-kube-api-access-zk4r9\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.009314 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.009279 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-metrics-certs\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.009499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.009397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4r9\" (UniqueName: \"kubernetes.io/projected/36db6e1f-6ebc-444a-89bf-4db7b32457ca-kube-api-access-zk4r9\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.009499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.009476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.010053 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:47.010032 2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 17 17:40:47.010158 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:47.010131 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert podName:36db6e1f-6ebc-444a-89bf-4db7b32457ca nodeName:}" failed. No retries permitted until 2026-09-17 17:40:47.510110727 +0000 UTC m=+584.072275566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert") pod "maas-controller-74ccbb7bf7-btjnv" (UID: "36db6e1f-6ebc-444a-89bf-4db7b32457ca") : secret "maas-controller-webhook-cert" not found Sep 17 17:40:47.012239 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.012220 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-metrics-certs\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.016849 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.016828 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b777b4495-bq469"] Sep 17 17:40:47.020008 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.019992 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.025409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.025388 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4r9\" (UniqueName: \"kubernetes.io/projected/36db6e1f-6ebc-444a-89bf-4db7b32457ca-kube-api-access-zk4r9\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.047123 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.047099 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b777b4495-bq469"] Sep 17 17:40:47.110589 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.110500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.110589 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.110571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-metrics-certs\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.110771 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.110604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/73c3b99c-fdac-4a97-b23a-39b928794969-kube-api-access-9xmtd\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.211187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.211154 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-metrics-certs\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.211187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.211191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/73c3b99c-fdac-4a97-b23a-39b928794969-kube-api-access-9xmtd\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.211400 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.211225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.211400 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:47.211320 2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 17 17:40:47.211400 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:47.211381 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert podName:73c3b99c-fdac-4a97-b23a-39b928794969 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:47.711362917 +0000 UTC m=+584.273527762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert") pod "maas-controller-6b777b4495-bq469" (UID: "73c3b99c-fdac-4a97-b23a-39b928794969") : secret "maas-controller-webhook-cert" not found Sep 17 17:40:47.213621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.213595 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-metrics-certs\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.221152 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.221129 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/73c3b99c-fdac-4a97-b23a-39b928794969-kube-api-access-9xmtd\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.513192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.513105 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:47.513352 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:47.513234 2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 17 17:40:47.513352 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:40:47.513300 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert podName:36db6e1f-6ebc-444a-89bf-4db7b32457ca nodeName:}" failed. No retries permitted until 2026-09-17 17:40:48.513281616 +0000 UTC m=+585.075446474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert") pod "maas-controller-74ccbb7bf7-btjnv" (UID: "36db6e1f-6ebc-444a-89bf-4db7b32457ca") : secret "maas-controller-webhook-cert" not found Sep 17 17:40:47.715444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.715414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.717777 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.717756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert\") pod \"maas-controller-6b777b4495-bq469\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:47.936208 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:47.936175 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:48.206832 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:48.206807 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b777b4495-bq469"] Sep 17 17:40:48.209649 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:40:48.209622 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c3b99c_fdac_4a97_b23a_39b928794969.slice/crio-5e25e61883436b70fc13963e38b6da3c901f8e1c772004303ad59cac74d59d3a WatchSource:0}: Error finding container 5e25e61883436b70fc13963e38b6da3c901f8e1c772004303ad59cac74d59d3a: Status 404 returned error can't find the container with id 5e25e61883436b70fc13963e38b6da3c901f8e1c772004303ad59cac74d59d3a Sep 17 17:40:48.337521 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:48.337485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-bq469" event={"ID":"73c3b99c-fdac-4a97-b23a-39b928794969","Type":"ContainerStarted","Data":"5e25e61883436b70fc13963e38b6da3c901f8e1c772004303ad59cac74d59d3a"} Sep 17 17:40:48.522598 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:48.522515 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:48.524940 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:48.524922 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") pod \"maas-controller-74ccbb7bf7-btjnv\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:48.646544 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:48.646505 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:48.923886 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:48.923724 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74ccbb7bf7-btjnv"] Sep 17 17:40:48.927014 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:40:48.926983 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36db6e1f_6ebc_444a_89bf_4db7b32457ca.slice/crio-48e64e35b46ecd943154d2294565ceaa1ae488f92af2c12c942d50e91b4a9848 WatchSource:0}: Error finding container 48e64e35b46ecd943154d2294565ceaa1ae488f92af2c12c942d50e91b4a9848: Status 404 returned error can't find the container with id 48e64e35b46ecd943154d2294565ceaa1ae488f92af2c12c942d50e91b4a9848 Sep 17 17:40:49.346427 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:49.346342 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" event={"ID":"36db6e1f-6ebc-444a-89bf-4db7b32457ca","Type":"ContainerStarted","Data":"48e64e35b46ecd943154d2294565ceaa1ae488f92af2c12c942d50e91b4a9848"} Sep 17 17:40:51.357012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:51.356973 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-bq469" event={"ID":"73c3b99c-fdac-4a97-b23a-39b928794969","Type":"ContainerStarted","Data":"0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02"} Sep 17 17:40:51.357469 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:51.357035 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:40:51.358631 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:51.358607 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" event={"ID":"36db6e1f-6ebc-444a-89bf-4db7b32457ca","Type":"ContainerStarted","Data":"b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e"} Sep 17 17:40:51.358748 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:51.358664 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:40:51.395559 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:51.395514 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b777b4495-bq469" podStartSLOduration=2.840820604 podStartE2EDuration="5.395497087s" totalImagesPullingTime="2.554676483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:51.393473152 +0000 UTC m=+587.955638013" watchObservedRunningTime="2026-09-17 17:40:51.395497087 +0000 UTC m=+587.957661992" Sep 17 17:40:51.429362 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:40:51.429315 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" podStartSLOduration=3.583165363 podStartE2EDuration="5.429300537s" totalImagesPullingTime="1.846135174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:51.427584022 +0000 UTC m=+587.989748884" watchObservedRunningTime="2026-09-17 17:40:51.429300537 +0000 UTC m=+587.991465396" Sep 17 17:41:02.364923 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.364888 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 17:41:02.365379 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.365071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:41:02.456088 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.456052 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74ccbb7bf7-btjnv"] Sep 17 17:41:02.456285 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.456260 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" podUID="36db6e1f-6ebc-444a-89bf-4db7b32457ca" containerName="manager" containerID="cri-o://b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e" gracePeriod=30 Sep 17 17:41:02.699852 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.699826 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:41:02.839583 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.839554 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk4r9\" (UniqueName: \"kubernetes.io/projected/36db6e1f-6ebc-444a-89bf-4db7b32457ca-kube-api-access-zk4r9\") pod \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " Sep 17 17:41:02.839583 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.839593 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") pod \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " Sep 17 17:41:02.839836 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.839626 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-metrics-certs\") pod \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\" (UID: \"36db6e1f-6ebc-444a-89bf-4db7b32457ca\") " Sep 17 17:41:02.841853 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.841817 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36db6e1f-6ebc-444a-89bf-4db7b32457ca-kube-api-access-zk4r9" (OuterVolumeSpecName: "kube-api-access-zk4r9") pod "36db6e1f-6ebc-444a-89bf-4db7b32457ca" (UID: "36db6e1f-6ebc-444a-89bf-4db7b32457ca"). InnerVolumeSpecName "kube-api-access-zk4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:41:02.841990 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.841849 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "36db6e1f-6ebc-444a-89bf-4db7b32457ca" (UID: "36db6e1f-6ebc-444a-89bf-4db7b32457ca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:41:02.841990 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.841909 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36db6e1f-6ebc-444a-89bf-4db7b32457ca" (UID: "36db6e1f-6ebc-444a-89bf-4db7b32457ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:41:02.941165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.941087 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4r9\" (UniqueName: \"kubernetes.io/projected/36db6e1f-6ebc-444a-89bf-4db7b32457ca-kube-api-access-zk4r9\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:41:02.941165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.941112 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-webhook-cert\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:41:02.941165 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:02.941122 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36db6e1f-6ebc-444a-89bf-4db7b32457ca-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:41:03.414004 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.413947 2566 generic.go:356] "Generic (PLEG): container finished" podID="36db6e1f-6ebc-444a-89bf-4db7b32457ca" containerID="b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e" exitCode=0 Sep 17 17:41:03.414004 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.413985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" event={"ID":"36db6e1f-6ebc-444a-89bf-4db7b32457ca","Type":"ContainerDied","Data":"b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e"} Sep 17 17:41:03.414511 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.414050 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" event={"ID":"36db6e1f-6ebc-444a-89bf-4db7b32457ca","Type":"ContainerDied","Data":"48e64e35b46ecd943154d2294565ceaa1ae488f92af2c12c942d50e91b4a9848"} Sep 17 17:41:03.414511 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.413997 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ccbb7bf7-btjnv" Sep 17 17:41:03.414511 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.414082 2566 scope.go:117] "RemoveContainer" containerID="b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e" Sep 17 17:41:03.422732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.422717 2566 scope.go:117] "RemoveContainer" containerID="b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e" Sep 17 17:41:03.423007 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:41:03.422988 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e\": container with ID starting with b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e not found: ID does not exist" containerID="b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e" Sep 17 17:41:03.423086 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.423014 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e"} err="failed to get container status \"b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e\": rpc error: code = NotFound desc = could not find container \"b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e\": container with ID starting with b0df6683fa95ba7b0c5db72bc23a066b38cc9985aebbc202236c95760ce2616e not found: ID does not exist" Sep 17 17:41:03.438501 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.438473 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74ccbb7bf7-btjnv"] Sep 17 17:41:03.442678 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:03.442660 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74ccbb7bf7-btjnv"] Sep 17 17:41:04.149674 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:04.149643 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36db6e1f-6ebc-444a-89bf-4db7b32457ca" path="/var/lib/kubelet/pods/36db6e1f-6ebc-444a-89bf-4db7b32457ca/volumes" Sep 17 17:41:25.297647 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.297612 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2"] Sep 17 17:41:25.298055 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.297904 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36db6e1f-6ebc-444a-89bf-4db7b32457ca" containerName="manager" Sep 17 17:41:25.298055 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.297914 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="36db6e1f-6ebc-444a-89bf-4db7b32457ca" containerName="manager" Sep 17 17:41:25.298055 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.297988 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="36db6e1f-6ebc-444a-89bf-4db7b32457ca" containerName="manager" Sep 17 17:41:25.302005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.301989 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.304289 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.304261 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 17:41:25.304409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.304363 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mzh7j\"" Sep 17 17:41:25.304599 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.304586 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 17:41:25.312794 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.312768 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2"] Sep 17 17:41:25.394432 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.394404 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-maas-api-tls\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.394600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.394463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.394600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.394482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6fb890-676a-4ec4-9e85-55864bf63eff-openshift-service-ca\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.394600 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.394511 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlw2l\" (UniqueName: \"kubernetes.io/projected/9e6fb890-676a-4ec4-9e85-55864bf63eff-kube-api-access-mlw2l\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.495122 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.495093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.495122 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.495121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6fb890-676a-4ec4-9e85-55864bf63eff-openshift-service-ca\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.495339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.495142 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlw2l\" (UniqueName: \"kubernetes.io/projected/9e6fb890-676a-4ec4-9e85-55864bf63eff-kube-api-access-mlw2l\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.495339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.495178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-maas-api-tls\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.495339 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:41:25.495264 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 17 17:41:25.495339 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:41:25.495332 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs podName:9e6fb890-676a-4ec4-9e85-55864bf63eff nodeName:}" failed. No retries permitted until 2026-09-17 17:41:25.995310423 +0000 UTC m=+622.557475280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs") pod "maas-api-6c99d599cf-5ngv2" (UID: "9e6fb890-676a-4ec4-9e85-55864bf63eff") : secret "maas-api-metrics-tls" not found Sep 17 17:41:25.495997 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.495973 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6fb890-676a-4ec4-9e85-55864bf63eff-openshift-service-ca\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.497675 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.497652 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-maas-api-tls\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.508006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.507980 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlw2l\" (UniqueName: \"kubernetes.io/projected/9e6fb890-676a-4ec4-9e85-55864bf63eff-kube-api-access-mlw2l\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.900731 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.900696 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-98qth"] Sep 17 17:41:25.902853 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.902839 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:25.905260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.905237 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xq52t\"" Sep 17 17:41:25.905375 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.905268 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 17:41:25.915486 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.915461 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-98qth"] Sep 17 17:41:25.997777 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.997748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:25.997940 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.997828 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp649\" (UniqueName: \"kubernetes.io/projected/93720089-403a-41aa-ae26-d739aee03bc3-kube-api-access-bp649\") pod \"payload-pre-processing-59fbc47bf8-98qth\" (UID: \"93720089-403a-41aa-ae26-d739aee03bc3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:25.997940 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:25.997854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93720089-403a-41aa-ae26-d739aee03bc3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-98qth\" (UID: \"93720089-403a-41aa-ae26-d739aee03bc3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:26.000428 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.000380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs\") pod \"maas-api-6c99d599cf-5ngv2\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:26.098382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.098345 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp649\" (UniqueName: \"kubernetes.io/projected/93720089-403a-41aa-ae26-d739aee03bc3-kube-api-access-bp649\") pod \"payload-pre-processing-59fbc47bf8-98qth\" (UID: \"93720089-403a-41aa-ae26-d739aee03bc3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:26.098382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.098383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93720089-403a-41aa-ae26-d739aee03bc3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-98qth\" (UID: \"93720089-403a-41aa-ae26-d739aee03bc3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:26.099107 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.099083 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93720089-403a-41aa-ae26-d739aee03bc3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-98qth\" (UID: \"93720089-403a-41aa-ae26-d739aee03bc3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:26.109878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.109850 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp649\" (UniqueName: \"kubernetes.io/projected/93720089-403a-41aa-ae26-d739aee03bc3-kube-api-access-bp649\") pod \"payload-pre-processing-59fbc47bf8-98qth\" (UID: \"93720089-403a-41aa-ae26-d739aee03bc3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:26.211323 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.211249 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:26.212760 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.212739 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:26.502120 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.501535 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-98qth"] Sep 17 17:41:26.504109 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:41:26.504075 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93720089_403a_41aa_ae26_d739aee03bc3.slice/crio-04353dc5e4be7581172b2cfec749993d9fc1477f247d8cf399578cde70f8e30e WatchSource:0}: Error finding container 04353dc5e4be7581172b2cfec749993d9fc1477f247d8cf399578cde70f8e30e: Status 404 returned error can't find the container with id 04353dc5e4be7581172b2cfec749993d9fc1477f247d8cf399578cde70f8e30e Sep 17 17:41:26.516292 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.516264 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2"] Sep 17 17:41:26.519669 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:26.519639 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" event={"ID":"93720089-403a-41aa-ae26-d739aee03bc3","Type":"ContainerStarted","Data":"04353dc5e4be7581172b2cfec749993d9fc1477f247d8cf399578cde70f8e30e"} Sep 17 17:41:26.519746 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:41:26.519687 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6fb890_676a_4ec4_9e85_55864bf63eff.slice/crio-d9222c8e1ab9d34e6bb815f412886e6250622a16f632b089ac9295a813e88dab WatchSource:0}: Error finding container d9222c8e1ab9d34e6bb815f412886e6250622a16f632b089ac9295a813e88dab: Status 404 returned error can't find the container with id d9222c8e1ab9d34e6bb815f412886e6250622a16f632b089ac9295a813e88dab Sep 17 17:41:27.526119 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:27.526084 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" event={"ID":"9e6fb890-676a-4ec4-9e85-55864bf63eff","Type":"ContainerStarted","Data":"d9222c8e1ab9d34e6bb815f412886e6250622a16f632b089ac9295a813e88dab"} Sep 17 17:41:28.531893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:28.531851 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" event={"ID":"93720089-403a-41aa-ae26-d739aee03bc3","Type":"ContainerStarted","Data":"4bdfc0cb67f8852901a90816ce7a8afde595b128168926956a37f99d0a0358a6"} Sep 17 17:41:28.532272 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:28.531979 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:28.566073 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:28.566026 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" podStartSLOduration=2.34222936 podStartE2EDuration="3.566009845s" totalImagesPullingTime="1.223780485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:28.565639619 +0000 UTC m=+625.127804480" watchObservedRunningTime="2026-09-17 17:41:28.566009845 +0000 UTC m=+625.128174704" Sep 17 17:41:29.539727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:29.539688 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" event={"ID":"9e6fb890-676a-4ec4-9e85-55864bf63eff","Type":"ContainerStarted","Data":"a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb"} Sep 17 17:41:29.558415 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:29.558364 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" podStartSLOduration=2.615024031 podStartE2EDuration="4.558344862s" totalImagesPullingTime="1.943320831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:29.558241336 +0000 UTC m=+626.120406208" watchObservedRunningTime="2026-09-17 17:41:29.558344862 +0000 UTC m=+626.120509724" Sep 17 17:41:30.544110 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:30.544078 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:36.553683 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:36.553656 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 17:41:39.541824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:39.541789 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-98qth" Sep 17 17:41:41.286533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.286502 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5"] Sep 17 17:41:41.289681 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.289660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.293840 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.293820 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 17:41:41.293979 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.293819 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 17 17:41:41.293979 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.293821 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hxzsq\"" Sep 17 17:41:41.293979 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.293856 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 17:41:41.305765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.305744 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5"] Sep 17 17:41:41.314421 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.314401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.314540 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.314433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwc6l\" (UniqueName: \"kubernetes.io/projected/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-kube-api-access-vwc6l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.314540 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.314455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.314540 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.314487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.314699 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.314552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.314699 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.314573 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.415833 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.415799 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.415833 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.415837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwc6l\" (UniqueName: \"kubernetes.io/projected/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-kube-api-access-vwc6l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416094 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.415856 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416094 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.415878 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416094 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.415904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416094 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.415920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416302 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.416269 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416356 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.416323 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.416356 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.416349 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.418212 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.418190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.418604 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.418584 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.425369 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.425351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwc6l\" (UniqueName: \"kubernetes.io/projected/0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7-kube-api-access-vwc6l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5\" (UID: \"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.600701 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.600671 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:41.873644 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:41.873613 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5"] Sep 17 17:41:41.876742 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:41:41.876712 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1dfeab_8ea2_4bbf_bc25_f72e83317eb7.slice/crio-94338171bff12c9ff9cbb6f5c06086344c8a53339938c94d8b445fce9536b34b WatchSource:0}: Error finding container 94338171bff12c9ff9cbb6f5c06086344c8a53339938c94d8b445fce9536b34b: Status 404 returned error can't find the container with id 94338171bff12c9ff9cbb6f5c06086344c8a53339938c94d8b445fce9536b34b Sep 17 17:41:42.595863 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:42.595820 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" event={"ID":"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7","Type":"ContainerStarted","Data":"94338171bff12c9ff9cbb6f5c06086344c8a53339938c94d8b445fce9536b34b"} Sep 17 17:41:43.602408 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:43.602375 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" event={"ID":"0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7","Type":"ContainerStarted","Data":"9c5b887c80194fb9b726921564937ad118d6fa8196379ee12041a32d8435e114"} Sep 17 17:41:43.622137 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:43.622090 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" podStartSLOduration=1.385810202 podStartE2EDuration="2.62207394s" totalImagesPullingTime="1.236263738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:43.620590665 +0000 UTC m=+640.182755525" watchObservedRunningTime="2026-09-17 17:41:43.62207394 +0000 UTC m=+640.184238800" Sep 17 17:41:48.334006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.333974 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx"] Sep 17 17:41:48.338114 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.338092 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.340900 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.340876 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 17 17:41:48.352568 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.352547 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx"] Sep 17 17:41:48.372901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.372875 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.373068 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.372907 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.373068 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.372937 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.373068 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.373042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.373194 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.373090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263642c-7d0e-47ec-9625-2057f27bdfbb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.373194 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.373130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r728m\" (UniqueName: \"kubernetes.io/projected/1263642c-7d0e-47ec-9625-2057f27bdfbb-kube-api-access-r728m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.473882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.473848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263642c-7d0e-47ec-9625-2057f27bdfbb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.473882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.473890 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r728m\" (UniqueName: \"kubernetes.io/projected/1263642c-7d0e-47ec-9625-2057f27bdfbb-kube-api-access-r728m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.473914 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.473938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.474010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474172 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.474050 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474415 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.474382 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474484 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.474395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.474567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.474549 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.476429 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.476393 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263642c-7d0e-47ec-9625-2057f27bdfbb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.476720 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.476702 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263642c-7d0e-47ec-9625-2057f27bdfbb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.481893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.481871 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r728m\" (UniqueName: \"kubernetes.io/projected/1263642c-7d0e-47ec-9625-2057f27bdfbb-kube-api-access-r728m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6tqgx\" (UID: \"1263642c-7d0e-47ec-9625-2057f27bdfbb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.648982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.648875 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:48.922937 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:48.922912 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx"] Sep 17 17:41:48.925612 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:41:48.925581 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1263642c_7d0e_47ec_9625_2057f27bdfbb.slice/crio-4df14c509e5eb0f7560f74553e72bac577335dee7744912810258815be6abe22 WatchSource:0}: Error finding container 4df14c509e5eb0f7560f74553e72bac577335dee7744912810258815be6abe22: Status 404 returned error can't find the container with id 4df14c509e5eb0f7560f74553e72bac577335dee7744912810258815be6abe22 Sep 17 17:41:49.634805 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:49.634768 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" event={"ID":"1263642c-7d0e-47ec-9625-2057f27bdfbb","Type":"ContainerStarted","Data":"d667b4ff58244b6864c052c6c34d55202859ab22e39553fea4a17b8076eaac1c"} Sep 17 17:41:49.635211 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:49.634842 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" event={"ID":"1263642c-7d0e-47ec-9625-2057f27bdfbb","Type":"ContainerStarted","Data":"4df14c509e5eb0f7560f74553e72bac577335dee7744912810258815be6abe22"} Sep 17 17:41:49.655591 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:49.655538 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" podStartSLOduration=1.469680517 podStartE2EDuration="1.655522129s" totalImagesPullingTime="185.841612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:49.654717741 +0000 UTC m=+646.216882602" watchObservedRunningTime="2026-09-17 17:41:49.655522129 +0000 UTC m=+646.217686989" Sep 17 17:41:51.601470 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:51.601437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:51.601824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:51.601506 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:51.613926 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:51.613903 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:51.654471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:51.654448 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5" Sep 17 17:41:58.649046 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:58.649001 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:58.649714 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:58.649081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:58.662472 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:58.662442 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:41:58.683950 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:41:58.683922 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6tqgx" Sep 17 17:42:51.642599 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.642568 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr"] Sep 17 17:42:51.646117 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.646099 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.652073 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.652048 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 17 17:42:51.664575 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.664550 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr"] Sep 17 17:42:51.683798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.683770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0773dd-19eb-47cd-929a-a4a691fef141-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.683903 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.683825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.683903 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.683848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.683903 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.683872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wgh\" (UniqueName: \"kubernetes.io/projected/1d0773dd-19eb-47cd-929a-a4a691fef141-kube-api-access-x2wgh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.684045 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.683905 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.684045 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.683918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785275 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785275 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785509 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wgh\" (UniqueName: \"kubernetes.io/projected/1d0773dd-19eb-47cd-929a-a4a691fef141-kube-api-access-x2wgh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785509 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785515 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785610 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0773dd-19eb-47cd-929a-a4a691fef141-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785747 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785727 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785804 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.785846 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.785826 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.787493 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.787459 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0773dd-19eb-47cd-929a-a4a691fef141-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.787891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.787873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0773dd-19eb-47cd-929a-a4a691fef141-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.794373 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.794341 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wgh\" (UniqueName: \"kubernetes.io/projected/1d0773dd-19eb-47cd-929a-a4a691fef141-kube-api-access-x2wgh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgxwr\" (UID: \"1d0773dd-19eb-47cd-929a-a4a691fef141\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:51.956870 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:51.956786 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:42:52.231525 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:52.231501 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr"] Sep 17 17:42:52.233137 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:42:52.233109 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0773dd_19eb_47cd_929a_a4a691fef141.slice/crio-fee1311a9c3a980b468e0c0fb69ff159854b246b3b3f134a85f3a9938cce75d0 WatchSource:0}: Error finding container fee1311a9c3a980b468e0c0fb69ff159854b246b3b3f134a85f3a9938cce75d0: Status 404 returned error can't find the container with id fee1311a9c3a980b468e0c0fb69ff159854b246b3b3f134a85f3a9938cce75d0 Sep 17 17:42:52.234428 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:52.234408 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:42:52.930135 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:52.930097 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" event={"ID":"1d0773dd-19eb-47cd-929a-a4a691fef141","Type":"ContainerStarted","Data":"f013d7d8441bb2d9bac942467b965382b0df807323150240f322de1f5bcd7fda"} Sep 17 17:42:52.930509 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:52.930158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" event={"ID":"1d0773dd-19eb-47cd-929a-a4a691fef141","Type":"ContainerStarted","Data":"fee1311a9c3a980b468e0c0fb69ff159854b246b3b3f134a85f3a9938cce75d0"} Sep 17 17:42:52.953475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:42:52.953430 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" podStartSLOduration=1.744778679 podStartE2EDuration="1.953411907s" totalImagesPullingTime="208.633228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:52.95289298 +0000 UTC m=+709.515057842" watchObservedRunningTime="2026-09-17 17:42:52.953411907 +0000 UTC m=+709.515576802" Sep 17 17:43:01.957281 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:01.957188 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:43:01.957281 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:01.957255 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:43:01.970887 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:01.970860 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:43:01.982405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:01.982381 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgxwr" Sep 17 17:43:05.130917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.130883 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk"] Sep 17 17:43:05.134330 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.134305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.137042 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.137017 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 17 17:43:05.146283 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.146263 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk"] Sep 17 17:43:05.193467 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.193413 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.193621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.193475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27wh\" (UniqueName: \"kubernetes.io/projected/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-kube-api-access-j27wh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.193621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.193584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.193621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.193612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.193747 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.193634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.193747 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.193712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.294533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.294533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27wh\" (UniqueName: \"kubernetes.io/projected/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-kube-api-access-j27wh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.294752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.294752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.294752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.294752 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.295004 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.294943 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.295088 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.295066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.295088 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.295081 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.296776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.296753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.297033 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.297009 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.304404 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.304377 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27wh\" (UniqueName: \"kubernetes.io/projected/58c000c9-0b8d-4d7c-9c5a-3a8fb986f549-kube-api-access-j27wh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk\" (UID: \"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.442872 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.442781 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:05.714255 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.714228 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk"] Sep 17 17:43:05.716208 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:43:05.716177 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58c000c9_0b8d_4d7c_9c5a_3a8fb986f549.slice/crio-1a086df108a55fb8f5651b9de680f36ecf1117063d28b9144ecbd67f4c437814 WatchSource:0}: Error finding container 1a086df108a55fb8f5651b9de680f36ecf1117063d28b9144ecbd67f4c437814: Status 404 returned error can't find the container with id 1a086df108a55fb8f5651b9de680f36ecf1117063d28b9144ecbd67f4c437814 Sep 17 17:43:05.990543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.990512 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" event={"ID":"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549","Type":"ContainerStarted","Data":"8f6a3d3c495def7bb8c61b3b3d5cd556a488d915d40a857818acd04a7383e96e"} Sep 17 17:43:05.990694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:05.990551 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" event={"ID":"58c000c9-0b8d-4d7c-9c5a-3a8fb986f549","Type":"ContainerStarted","Data":"1a086df108a55fb8f5651b9de680f36ecf1117063d28b9144ecbd67f4c437814"} Sep 17 17:43:06.009134 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:06.009079 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" podStartSLOduration=0.80521285 podStartE2EDuration="1.009062799s" totalImagesPullingTime="203.849949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:06.007520054 +0000 UTC m=+722.569684896" watchObservedRunningTime="2026-09-17 17:43:06.009062799 +0000 UTC m=+722.571227658" Sep 17 17:43:15.443359 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:15.443322 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:15.443868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:15.443375 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:15.455762 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:15.455732 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:43:16.054383 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:43:16.054355 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk" Sep 17 17:44:41.236358 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.236311 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55674c7f-psqxl"] Sep 17 17:44:41.236870 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.236525 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55674c7f-psqxl" podUID="69665146-80d3-49cf-b91c-eaf8149581e3" containerName="authorino" containerID="cri-o://f9243440ee21c81562f8f3d2445e9fec00446fdf0be776ea9b9bc32651a17b52" gracePeriod=30 Sep 17 17:44:41.433845 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.433814 2566 generic.go:356] "Generic (PLEG): container finished" podID="69665146-80d3-49cf-b91c-eaf8149581e3" containerID="f9243440ee21c81562f8f3d2445e9fec00446fdf0be776ea9b9bc32651a17b52" exitCode=0 Sep 17 17:44:41.434017 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.433896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55674c7f-psqxl" event={"ID":"69665146-80d3-49cf-b91c-eaf8149581e3","Type":"ContainerDied","Data":"f9243440ee21c81562f8f3d2445e9fec00446fdf0be776ea9b9bc32651a17b52"} Sep 17 17:44:41.489390 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.489337 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:44:41.579035 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.579004 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7pp2\" (UniqueName: \"kubernetes.io/projected/69665146-80d3-49cf-b91c-eaf8149581e3-kube-api-access-p7pp2\") pod \"69665146-80d3-49cf-b91c-eaf8149581e3\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " Sep 17 17:44:41.579181 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.579045 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69665146-80d3-49cf-b91c-eaf8149581e3-tls-cert\") pod \"69665146-80d3-49cf-b91c-eaf8149581e3\" (UID: \"69665146-80d3-49cf-b91c-eaf8149581e3\") " Sep 17 17:44:41.581178 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.581145 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69665146-80d3-49cf-b91c-eaf8149581e3-kube-api-access-p7pp2" (OuterVolumeSpecName: "kube-api-access-p7pp2") pod "69665146-80d3-49cf-b91c-eaf8149581e3" (UID: "69665146-80d3-49cf-b91c-eaf8149581e3"). InnerVolumeSpecName "kube-api-access-p7pp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:44:41.588827 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.588799 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69665146-80d3-49cf-b91c-eaf8149581e3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "69665146-80d3-49cf-b91c-eaf8149581e3" (UID: "69665146-80d3-49cf-b91c-eaf8149581e3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:44:41.679566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.679537 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7pp2\" (UniqueName: \"kubernetes.io/projected/69665146-80d3-49cf-b91c-eaf8149581e3-kube-api-access-p7pp2\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:44:41.679566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:41.679562 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69665146-80d3-49cf-b91c-eaf8149581e3-tls-cert\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:44:42.439849 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:42.439767 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55674c7f-psqxl" event={"ID":"69665146-80d3-49cf-b91c-eaf8149581e3","Type":"ContainerDied","Data":"5a0cb566bedabb56201261ed5030e822ad14988f75a563610e7c2f865a36aa4c"} Sep 17 17:44:42.439849 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:42.439816 2566 scope.go:117] "RemoveContainer" containerID="f9243440ee21c81562f8f3d2445e9fec00446fdf0be776ea9b9bc32651a17b52" Sep 17 17:44:42.440288 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:42.439780 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55674c7f-psqxl" Sep 17 17:44:42.458662 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:42.458638 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55674c7f-psqxl"] Sep 17 17:44:42.465917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:42.465891 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55674c7f-psqxl"] Sep 17 17:44:44.149358 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:44:44.149329 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69665146-80d3-49cf-b91c-eaf8149581e3" path="/var/lib/kubelet/pods/69665146-80d3-49cf-b91c-eaf8149581e3/volumes" Sep 17 17:45:01.785861 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.785825 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-5fbf2a"] Sep 17 17:45:01.786260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.786168 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69665146-80d3-49cf-b91c-eaf8149581e3" containerName="authorino" Sep 17 17:45:01.786260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.786180 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69665146-80d3-49cf-b91c-eaf8149581e3" containerName="authorino" Sep 17 17:45:01.786260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.786257 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69665146-80d3-49cf-b91c-eaf8149581e3" containerName="authorino" Sep 17 17:45:01.788980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.788944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:01.796685 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.796662 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z7xqm\"" Sep 17 17:45:01.809016 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.808992 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-5fbf2a"] Sep 17 17:45:01.851127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.851095 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnktl\" (UniqueName: \"kubernetes.io/projected/2e476b09-419c-47b1-9128-66cd55e11af3-kube-api-access-gnktl\") pod \"test-curl-7612-5fbf2a\" (UID: \"2e476b09-419c-47b1-9128-66cd55e11af3\") " pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:01.952546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.952509 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnktl\" (UniqueName: \"kubernetes.io/projected/2e476b09-419c-47b1-9128-66cd55e11af3-kube-api-access-gnktl\") pod \"test-curl-7612-5fbf2a\" (UID: \"2e476b09-419c-47b1-9128-66cd55e11af3\") " pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:01.962869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:01.962841 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnktl\" (UniqueName: \"kubernetes.io/projected/2e476b09-419c-47b1-9128-66cd55e11af3-kube-api-access-gnktl\") pod \"test-curl-7612-5fbf2a\" (UID: \"2e476b09-419c-47b1-9128-66cd55e11af3\") " pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:02.097847 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:02.097814 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:02.390847 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:02.390823 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-5fbf2a"] Sep 17 17:45:02.392568 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:02.392542 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e476b09_419c_47b1_9128_66cd55e11af3.slice/crio-05cd369452ccf835b50c0c9cc79b9f4e13c535b1387010559ad0be11a0406be5 WatchSource:0}: Error finding container 05cd369452ccf835b50c0c9cc79b9f4e13c535b1387010559ad0be11a0406be5: Status 404 returned error can't find the container with id 05cd369452ccf835b50c0c9cc79b9f4e13c535b1387010559ad0be11a0406be5 Sep 17 17:45:02.529981 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:02.529925 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-5fbf2a" event={"ID":"2e476b09-419c-47b1-9128-66cd55e11af3","Type":"ContainerStarted","Data":"05cd369452ccf835b50c0c9cc79b9f4e13c535b1387010559ad0be11a0406be5"} Sep 17 17:45:05.445434 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.445401 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l"] Sep 17 17:45:05.449818 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.449796 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.452255 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.452233 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-54966d-serving-cert\"" Sep 17 17:45:05.474868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.474840 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l"] Sep 17 17:45:05.586399 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.586364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543d4b92-7d49-4846-8b82-beb577a61dff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.586545 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.586424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-metrics-certs\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.586545 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.586491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtjkt\" (UniqueName: \"kubernetes.io/projected/543d4b92-7d49-4846-8b82-beb577a61dff-kube-api-access-gtjkt\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.586621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.586554 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-maas-api-tls\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.687919 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.687882 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543d4b92-7d49-4846-8b82-beb577a61dff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.688129 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.687938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-metrics-certs\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.688129 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.688069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtjkt\" (UniqueName: \"kubernetes.io/projected/543d4b92-7d49-4846-8b82-beb577a61dff-kube-api-access-gtjkt\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.688248 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.688141 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-maas-api-tls\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.688648 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.688624 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543d4b92-7d49-4846-8b82-beb577a61dff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.690304 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.690279 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-metrics-certs\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.690395 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.690317 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-maas-api-tls\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.709000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.708934 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtjkt\" (UniqueName: \"kubernetes.io/projected/543d4b92-7d49-4846-8b82-beb577a61dff-kube-api-access-gtjkt\") pod \"maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:05.759877 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:05.759841 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:06.045443 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.045413 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l"] Sep 17 17:45:06.046760 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:06.046730 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543d4b92_7d49_4846_8b82_beb577a61dff.slice/crio-be501a2d802c5915df4c5ebab0771785f510bce1559e3e7c66071d4901c9daa7 WatchSource:0}: Error finding container be501a2d802c5915df4c5ebab0771785f510bce1559e3e7c66071d4901c9daa7: Status 404 returned error can't find the container with id be501a2d802c5915df4c5ebab0771785f510bce1559e3e7c66071d4901c9daa7 Sep 17 17:45:06.552435 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.552374 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" event={"ID":"543d4b92-7d49-4846-8b82-beb577a61dff","Type":"ContainerStarted","Data":"10e436589786d0d10f124c91849a3102097f59598e6a37acf8339510b51adea5"} Sep 17 17:45:06.552825 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.552452 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" event={"ID":"543d4b92-7d49-4846-8b82-beb577a61dff","Type":"ContainerStarted","Data":"be501a2d802c5915df4c5ebab0771785f510bce1559e3e7c66071d4901c9daa7"} Sep 17 17:45:06.552825 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.552488 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:06.593170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.593116 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" podStartSLOduration=1.196942297 podStartE2EDuration="1.593099867s" totalImagesPullingTime="396.15757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:06.592242022 +0000 UTC m=+843.154406876" watchObservedRunningTime="2026-09-17 17:45:06.593099867 +0000 UTC m=+843.155264726" Sep 17 17:45:06.817889 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.817850 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq"] Sep 17 17:45:06.822225 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.822191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:06.829316 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.829289 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-54966d-dockercfg-f5ctb\"" Sep 17 17:45:06.829919 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.829858 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-54966d\"" Sep 17 17:45:06.851902 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.851865 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq"] Sep 17 17:45:06.900327 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.900290 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhf28\" (UniqueName: \"kubernetes.io/projected/1efb262c-45d6-46d5-b805-93af12aaca69-kube-api-access-bhf28\") pod \"payload-processing-e2e-inf-a-54966d-957c6685-8mdpq\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:06.900478 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:06.900364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efb262c-45d6-46d5-b805-93af12aaca69-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-54966d-957c6685-8mdpq\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.001243 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.001206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhf28\" (UniqueName: \"kubernetes.io/projected/1efb262c-45d6-46d5-b805-93af12aaca69-kube-api-access-bhf28\") pod \"payload-processing-e2e-inf-a-54966d-957c6685-8mdpq\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.001423 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.001290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efb262c-45d6-46d5-b805-93af12aaca69-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-54966d-957c6685-8mdpq\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.001949 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.001930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efb262c-45d6-46d5-b805-93af12aaca69-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-54966d-957c6685-8mdpq\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.016656 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.016621 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhf28\" (UniqueName: \"kubernetes.io/projected/1efb262c-45d6-46d5-b805-93af12aaca69-kube-api-access-bhf28\") pod \"payload-processing-e2e-inf-a-54966d-957c6685-8mdpq\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.134363 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.134283 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.443515 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.443490 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq"] Sep 17 17:45:07.445693 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:07.445667 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efb262c_45d6_46d5_b805_93af12aaca69.slice/crio-46a2ad31901d46d572303b70cf9591d31a74e1471d74ef34718484be8cf99da2 WatchSource:0}: Error finding container 46a2ad31901d46d572303b70cf9591d31a74e1471d74ef34718484be8cf99da2: Status 404 returned error can't find the container with id 46a2ad31901d46d572303b70cf9591d31a74e1471d74ef34718484be8cf99da2 Sep 17 17:45:07.559492 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.559452 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" event={"ID":"1efb262c-45d6-46d5-b805-93af12aaca69","Type":"ContainerStarted","Data":"4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996"} Sep 17 17:45:07.559862 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.559518 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" event={"ID":"1efb262c-45d6-46d5-b805-93af12aaca69","Type":"ContainerStarted","Data":"46a2ad31901d46d572303b70cf9591d31a74e1471d74ef34718484be8cf99da2"} Sep 17 17:45:07.559862 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.559548 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:07.591813 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:07.591748 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" podStartSLOduration=1.5917261360000001 podStartE2EDuration="1.591726136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:07.591124675 +0000 UTC m=+844.153289538" watchObservedRunningTime="2026-09-17 17:45:07.591726136 +0000 UTC m=+844.153890997" Sep 17 17:45:08.967940 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:08.967905 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6"] Sep 17 17:45:08.971818 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:08.971795 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:08.974413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:08.974392 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8aeea1-serving-cert\"" Sep 17 17:45:08.985171 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:08.985146 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6"] Sep 17 17:45:09.119087 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.119052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-metrics-certs\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.119222 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.119119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567dc582-bcdd-482b-9dc2-3e747952d4d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.119222 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.119145 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p649q\" (UniqueName: \"kubernetes.io/projected/567dc582-bcdd-482b-9dc2-3e747952d4d2-kube-api-access-p649q\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.119222 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.119169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.219834 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.219755 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-metrics-certs\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.219834 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.219827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567dc582-bcdd-482b-9dc2-3e747952d4d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.220095 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.219872 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p649q\" (UniqueName: \"kubernetes.io/projected/567dc582-bcdd-482b-9dc2-3e747952d4d2-kube-api-access-p649q\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.220095 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.219894 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.220095 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:09.220073 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-serving-cert: secret "maas-api-e2e-worker-w4-8aeea1-serving-cert" not found Sep 17 17:45:09.220261 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:09.220150 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls podName:567dc582-bcdd-482b-9dc2-3e747952d4d2 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:09.720126085 +0000 UTC m=+846.282290925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls") pod "maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" (UID: "567dc582-bcdd-482b-9dc2-3e747952d4d2") : secret "maas-api-e2e-worker-w4-8aeea1-serving-cert" not found Sep 17 17:45:09.220637 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.220620 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567dc582-bcdd-482b-9dc2-3e747952d4d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.222438 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.222406 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-metrics-certs\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.232593 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.232573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p649q\" (UniqueName: \"kubernetes.io/projected/567dc582-bcdd-482b-9dc2-3e747952d4d2-kube-api-access-p649q\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.725543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.725505 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.728014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.727982 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:09.882890 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:09.882850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:10.159310 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:10.159279 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6"] Sep 17 17:45:10.162823 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:10.162788 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567dc582_bcdd_482b_9dc2_3e747952d4d2.slice/crio-60dc1a2c5d1a9e18d41ff95a4cf3d1d6b79bbc4955a1a4a829243f9723593925 WatchSource:0}: Error finding container 60dc1a2c5d1a9e18d41ff95a4cf3d1d6b79bbc4955a1a4a829243f9723593925: Status 404 returned error can't find the container with id 60dc1a2c5d1a9e18d41ff95a4cf3d1d6b79bbc4955a1a4a829243f9723593925 Sep 17 17:45:10.576550 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:10.576519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" event={"ID":"567dc582-bcdd-482b-9dc2-3e747952d4d2","Type":"ContainerStarted","Data":"60dc1a2c5d1a9e18d41ff95a4cf3d1d6b79bbc4955a1a4a829243f9723593925"} Sep 17 17:45:11.582825 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:11.582784 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" event={"ID":"567dc582-bcdd-482b-9dc2-3e747952d4d2","Type":"ContainerStarted","Data":"4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762"} Sep 17 17:45:11.583310 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:11.583005 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:11.633176 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:11.633130 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" podStartSLOduration=3.278750645 podStartE2EDuration="3.633114326s" totalImagesPullingTime="354.363681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:11.631222375 +0000 UTC m=+848.193387236" watchObservedRunningTime="2026-09-17 17:45:11.633114326 +0000 UTC m=+848.195279186" Sep 17 17:45:12.566401 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:12.566368 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:45:14.598757 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:14.598725 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-5fbf2a" event={"ID":"2e476b09-419c-47b1-9128-66cd55e11af3","Type":"ContainerStarted","Data":"2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2"} Sep 17 17:45:14.625243 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:14.625197 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7612-5fbf2a" podStartSLOduration=2.108637753 podStartE2EDuration="13.625181445s" totalImagesPullingTime="11.516543692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:14.624272549 +0000 UTC m=+851.186437409" watchObservedRunningTime="2026-09-17 17:45:14.625181445 +0000 UTC m=+851.187346354" Sep 17 17:45:15.675598 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.675562 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-5fbf2a"] Sep 17 17:45:15.752382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.752348 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-5fbf2a"] Sep 17 17:45:15.861453 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.861421 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-3fdaeb"] Sep 17 17:45:15.861808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.861783 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e476b09-419c-47b1-9128-66cd55e11af3" containerName="test-curl-7612-5fbf2a" Sep 17 17:45:15.861808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.861800 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e476b09-419c-47b1-9128-66cd55e11af3" containerName="test-curl-7612-5fbf2a" Sep 17 17:45:15.861981 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.861863 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e476b09-419c-47b1-9128-66cd55e11af3" containerName="test-curl-7612-5fbf2a" Sep 17 17:45:15.864936 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.864917 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:15.876699 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.876661 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-3fdaeb"] Sep 17 17:45:15.983711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:15.983639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfhq\" (UniqueName: \"kubernetes.io/projected/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd-kube-api-access-vdfhq\") pod \"test-curl-7612-3fdaeb\" (UID: \"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd\") " pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:16.084851 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.084812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfhq\" (UniqueName: \"kubernetes.io/projected/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd-kube-api-access-vdfhq\") pod \"test-curl-7612-3fdaeb\" (UID: \"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd\") " pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:16.093497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.093463 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfhq\" (UniqueName: \"kubernetes.io/projected/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd-kube-api-access-vdfhq\") pod \"test-curl-7612-3fdaeb\" (UID: \"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd\") " pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:16.174123 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.174087 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:16.451339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.451312 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-3fdaeb"] Sep 17 17:45:16.453694 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:16.453668 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbcb79cf_d42f_4f09_ad39_b3fc30a335fd.slice/crio-6da4772f940b6af77c692b035fe3aa7a265d9c5ad06e68da7e3cbdff0d90fc60 WatchSource:0}: Error finding container 6da4772f940b6af77c692b035fe3aa7a265d9c5ad06e68da7e3cbdff0d90fc60: Status 404 returned error can't find the container with id 6da4772f940b6af77c692b035fe3aa7a265d9c5ad06e68da7e3cbdff0d90fc60 Sep 17 17:45:16.609996 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.609930 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-3fdaeb" event={"ID":"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd","Type":"ContainerStarted","Data":"aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4"} Sep 17 17:45:16.610178 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.610013 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-3fdaeb" event={"ID":"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd","Type":"ContainerStarted","Data":"6da4772f940b6af77c692b035fe3aa7a265d9c5ad06e68da7e3cbdff0d90fc60"} Sep 17 17:45:16.610178 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.610012 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7612-5fbf2a" podUID="2e476b09-419c-47b1-9128-66cd55e11af3" containerName="test-curl-7612-5fbf2a" containerID="cri-o://2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2" gracePeriod=2 Sep 17 17:45:16.640309 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:16.640256 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7612-3fdaeb" podStartSLOduration=1.640235379 podStartE2EDuration="1.640235379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:16.637375924 +0000 UTC m=+853.199540784" watchObservedRunningTime="2026-09-17 17:45:16.640235379 +0000 UTC m=+853.202400240" Sep 17 17:45:17.469867 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:17.469774 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-3fdaeb"] Sep 17 17:45:17.471467 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:17.471442 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-3fdaeb"] Sep 17 17:45:17.594144 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:17.594120 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:45:17.614936 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:17.614900 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7612-3fdaeb" container="test-curl-7612-3fdaeb" Sep 17 17:45:18.567942 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:18.567904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:45:18.621067 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:18.620938 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7612-3fdaeb" podUID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" containerName="test-curl-7612-3fdaeb" containerID="cri-o://aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4" gracePeriod=2 Sep 17 17:45:18.784072 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:18.784046 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:18.911095 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:18.911002 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnktl\" (UniqueName: \"kubernetes.io/projected/2e476b09-419c-47b1-9128-66cd55e11af3-kube-api-access-gnktl\") pod \"2e476b09-419c-47b1-9128-66cd55e11af3\" (UID: \"2e476b09-419c-47b1-9128-66cd55e11af3\") " Sep 17 17:45:18.913107 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:18.913077 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e476b09-419c-47b1-9128-66cd55e11af3-kube-api-access-gnktl" (OuterVolumeSpecName: "kube-api-access-gnktl") pod "2e476b09-419c-47b1-9128-66cd55e11af3" (UID: "2e476b09-419c-47b1-9128-66cd55e11af3"). InnerVolumeSpecName "kube-api-access-gnktl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:45:19.012442 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.012404 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnktl\" (UniqueName: \"kubernetes.io/projected/2e476b09-419c-47b1-9128-66cd55e11af3-kube-api-access-gnktl\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:19.628221 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.628186 2566 generic.go:356] "Generic (PLEG): container finished" podID="2e476b09-419c-47b1-9128-66cd55e11af3" containerID="2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2" exitCode=137 Sep 17 17:45:19.628705 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.628235 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:19.628705 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.628282 2566 scope.go:117] "RemoveContainer" containerID="2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2" Sep 17 17:45:19.628705 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.628694 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h"] Sep 17 17:45:19.629390 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.629356 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" containerName="test-curl-7612-3fdaeb" Sep 17 17:45:19.629390 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.629378 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" containerName="test-curl-7612-3fdaeb" Sep 17 17:45:19.629570 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.629471 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" containerName="test-curl-7612-3fdaeb" Sep 17 17:45:19.632845 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.632826 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.635803 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.635781 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f7d735f2-serving-cert\"" Sep 17 17:45:19.638684 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.638666 2566 scope.go:117] "RemoveContainer" containerID="2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2" Sep 17 17:45:19.639051 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:19.639024 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2\": container with ID starting with 2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2 not found: ID does not exist" containerID="2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2" Sep 17 17:45:19.639163 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.639062 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2"} err="failed to get container status \"2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2\": rpc error: code = NotFound desc = could not find container \"2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2\": container with ID starting with 2d1f0b061c2965a5dab052624ef75cbdeff3cbfc5780fe32dac00df9e4aca1a2 not found: ID does not exist" Sep 17 17:45:19.660842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.660807 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h"] Sep 17 17:45:19.694014 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:19.693979 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-5fbf2a\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="2e476b09-419c-47b1-9128-66cd55e11af3" pod="opendatahub/test-curl-7612-5fbf2a" Sep 17 17:45:19.819557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.819518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94m7c\" (UniqueName: \"kubernetes.io/projected/ac32abfe-5c80-494e-b45e-1cb820531201-kube-api-access-94m7c\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.819557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.819567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac32abfe-5c80-494e-b45e-1cb820531201-openshift-service-ca\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.819778 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.819586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.819778 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.819602 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-metrics-certs\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.920994 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.920879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94m7c\" (UniqueName: \"kubernetes.io/projected/ac32abfe-5c80-494e-b45e-1cb820531201-kube-api-access-94m7c\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.920994 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.920924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac32abfe-5c80-494e-b45e-1cb820531201-openshift-service-ca\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.920994 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.920945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.920994 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.920989 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-metrics-certs\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.921288 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:19.921070 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-serving-cert: secret "maas-api-e2e-ait-f7d735f2-serving-cert" not found Sep 17 17:45:19.921288 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:19.921133 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls podName:ac32abfe-5c80-494e-b45e-1cb820531201 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:20.42111662 +0000 UTC m=+856.983281458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls") pod "maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" (UID: "ac32abfe-5c80-494e-b45e-1cb820531201") : secret "maas-api-e2e-ait-f7d735f2-serving-cert" not found Sep 17 17:45:19.921715 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.921695 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac32abfe-5c80-494e-b45e-1cb820531201-openshift-service-ca\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.923515 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.923495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-metrics-certs\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:19.936534 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:19.936508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94m7c\" (UniqueName: \"kubernetes.io/projected/ac32abfe-5c80-494e-b45e-1cb820531201-kube-api-access-94m7c\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:20.149913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.149882 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e476b09-419c-47b1-9128-66cd55e11af3" path="/var/lib/kubelet/pods/2e476b09-419c-47b1-9128-66cd55e11af3/volumes" Sep 17 17:45:20.427074 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.427037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:20.429550 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.429525 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls\") pod \"maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:20.546121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.546080 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:20.560247 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.560214 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr"] Sep 17 17:45:20.564841 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.564818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.570219 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.570193 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f7d735f2-dockercfg-xdnxn\"" Sep 17 17:45:20.570219 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.570203 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f7d735f2\"" Sep 17 17:45:20.578569 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.578536 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr"] Sep 17 17:45:20.646845 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.646815 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5"] Sep 17 17:45:20.650837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.650818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.679182 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.679099 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5"] Sep 17 17:45:20.729555 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.729521 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rj2j\" (UniqueName: \"kubernetes.io/projected/b5802f0e-0942-4496-91fd-f99d68e370aa-kube-api-access-5rj2j\") pod \"payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.729731 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.729640 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5802f0e-0942-4496-91fd-f99d68e370aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.767743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.767714 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:20.830819 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.830786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rj2j\" (UniqueName: \"kubernetes.io/projected/b5802f0e-0942-4496-91fd-f99d68e370aa-kube-api-access-5rj2j\") pod \"payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.831010 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.830848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f1fe87-393e-40cc-b509-a9697a6b3a32-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.831010 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.830876 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8t2\" (UniqueName: \"kubernetes.io/projected/f0f1fe87-393e-40cc-b509-a9697a6b3a32-kube-api-access-lz8t2\") pod \"payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.831010 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.830904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5802f0e-0942-4496-91fd-f99d68e370aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.831690 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.831664 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5802f0e-0942-4496-91fd-f99d68e370aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.839790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.839767 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h"] Sep 17 17:45:20.843061 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:20.843037 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac32abfe_5c80_494e_b45e_1cb820531201.slice/crio-6987f55d53c2e66bbf6fcfe74f509de7fc8873a293ef9820d3f0cfb341ea3ca6 WatchSource:0}: Error finding container 6987f55d53c2e66bbf6fcfe74f509de7fc8873a293ef9820d3f0cfb341ea3ca6: Status 404 returned error can't find the container with id 6987f55d53c2e66bbf6fcfe74f509de7fc8873a293ef9820d3f0cfb341ea3ca6 Sep 17 17:45:20.845352 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.845334 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rj2j\" (UniqueName: \"kubernetes.io/projected/b5802f0e-0942-4496-91fd-f99d68e370aa-kube-api-access-5rj2j\") pod \"payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.884555 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.884524 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:20.931864 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.931748 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdfhq\" (UniqueName: \"kubernetes.io/projected/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd-kube-api-access-vdfhq\") pod \"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd\" (UID: \"fbcb79cf-d42f-4f09-ad39-b3fc30a335fd\") " Sep 17 17:45:20.931864 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.931855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f1fe87-393e-40cc-b509-a9697a6b3a32-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.932127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.931888 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8t2\" (UniqueName: \"kubernetes.io/projected/f0f1fe87-393e-40cc-b509-a9697a6b3a32-kube-api-access-lz8t2\") pod \"payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.933938 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.933300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f1fe87-393e-40cc-b509-a9697a6b3a32-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.934335 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.934300 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd-kube-api-access-vdfhq" (OuterVolumeSpecName: "kube-api-access-vdfhq") pod "fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" (UID: "fbcb79cf-d42f-4f09-ad39-b3fc30a335fd"). InnerVolumeSpecName "kube-api-access-vdfhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:45:20.945287 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.945079 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8t2\" (UniqueName: \"kubernetes.io/projected/f0f1fe87-393e-40cc-b509-a9697a6b3a32-kube-api-access-lz8t2\") pod \"payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:20.995031 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:20.994997 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:21.033419 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.033384 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfhq\" (UniqueName: \"kubernetes.io/projected/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd-kube-api-access-vdfhq\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:21.195588 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.195561 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr"] Sep 17 17:45:21.203063 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:21.203033 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5802f0e_0942_4496_91fd_f99d68e370aa.slice/crio-1f4f4b53943887acad7398bce2c358f4a66417281002c27ebbcb74688e2b8187 WatchSource:0}: Error finding container 1f4f4b53943887acad7398bce2c358f4a66417281002c27ebbcb74688e2b8187: Status 404 returned error can't find the container with id 1f4f4b53943887acad7398bce2c358f4a66417281002c27ebbcb74688e2b8187 Sep 17 17:45:21.321571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.321366 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5"] Sep 17 17:45:21.323843 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:21.323812 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f1fe87_393e_40cc_b509_a9697a6b3a32.slice/crio-ab855bfa0dea7e1fc627691fc685553bb488714b79667e3541ac7b114cf7a88a WatchSource:0}: Error finding container ab855bfa0dea7e1fc627691fc685553bb488714b79667e3541ac7b114cf7a88a: Status 404 returned error can't find the container with id ab855bfa0dea7e1fc627691fc685553bb488714b79667e3541ac7b114cf7a88a Sep 17 17:45:21.643698 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.643658 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" event={"ID":"ac32abfe-5c80-494e-b45e-1cb820531201","Type":"ContainerStarted","Data":"afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144"} Sep 17 17:45:21.643967 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.643719 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" event={"ID":"ac32abfe-5c80-494e-b45e-1cb820531201","Type":"ContainerStarted","Data":"6987f55d53c2e66bbf6fcfe74f509de7fc8873a293ef9820d3f0cfb341ea3ca6"} Sep 17 17:45:21.643967 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.643750 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:21.645184 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.645149 2566 generic.go:356] "Generic (PLEG): container finished" podID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" containerID="aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4" exitCode=137 Sep 17 17:45:21.645184 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.645179 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:21.645337 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.645255 2566 scope.go:117] "RemoveContainer" containerID="aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4" Sep 17 17:45:21.647466 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.647416 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" event={"ID":"f0f1fe87-393e-40cc-b509-a9697a6b3a32","Type":"ContainerStarted","Data":"c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6"} Sep 17 17:45:21.647466 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.647465 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" event={"ID":"f0f1fe87-393e-40cc-b509-a9697a6b3a32","Type":"ContainerStarted","Data":"ab855bfa0dea7e1fc627691fc685553bb488714b79667e3541ac7b114cf7a88a"} Sep 17 17:45:21.647922 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.647612 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:21.649369 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.649346 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" event={"ID":"b5802f0e-0942-4496-91fd-f99d68e370aa","Type":"ContainerStarted","Data":"8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a"} Sep 17 17:45:21.649471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.649389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" event={"ID":"b5802f0e-0942-4496-91fd-f99d68e370aa","Type":"ContainerStarted","Data":"1f4f4b53943887acad7398bce2c358f4a66417281002c27ebbcb74688e2b8187"} Sep 17 17:45:21.649528 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.649514 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:21.655392 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.655376 2566 scope.go:117] "RemoveContainer" containerID="aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4" Sep 17 17:45:21.655700 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:21.655681 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4\": container with ID starting with aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4 not found: ID does not exist" containerID="aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4" Sep 17 17:45:21.655751 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.655715 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4"} err="failed to get container status \"aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4\": rpc error: code = NotFound desc = could not find container \"aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4\": container with ID starting with aa74cccfbd669a86143aec922cfe44ecf3b72ec9d7926e98ea01920a3db7cbc4 not found: ID does not exist" Sep 17 17:45:21.670139 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:21.670115 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-3fdaeb\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" pod="opendatahub/test-curl-7612-3fdaeb" Sep 17 17:45:21.674014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.673935 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" podStartSLOduration=2.344440874 podStartE2EDuration="2.673918193s" totalImagesPullingTime="329.477319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:21.667950257 +0000 UTC m=+858.230115119" watchObservedRunningTime="2026-09-17 17:45:21.673918193 +0000 UTC m=+858.236083053" Sep 17 17:45:21.708833 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.708777 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" podStartSLOduration=1.708757759 podStartE2EDuration="1.708757759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:21.707540259 +0000 UTC m=+858.269705120" watchObservedRunningTime="2026-09-17 17:45:21.708757759 +0000 UTC m=+858.270922625" Sep 17 17:45:21.737609 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:21.737538 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" podStartSLOduration=1.737520612 podStartE2EDuration="1.737520612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:21.737470668 +0000 UTC m=+858.299635553" watchObservedRunningTime="2026-09-17 17:45:21.737520612 +0000 UTC m=+858.299685485" Sep 17 17:45:22.151174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:22.151134 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbcb79cf-d42f-4f09-ad39-b3fc30a335fd" path="/var/lib/kubelet/pods/fbcb79cf-d42f-4f09-ad39-b3fc30a335fd/volumes" Sep 17 17:45:27.668494 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:27.668454 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:29.284104 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.284074 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6"] Sep 17 17:45:29.291765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.291747 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.294862 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.294841 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-af14f8\"" Sep 17 17:45:29.295006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.294903 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-af14f8-dockercfg-6tb6q\"" Sep 17 17:45:29.301998 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.301974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d176e8-6196-4867-a79b-b78b51101671-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.302095 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.302050 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954vk\" (UniqueName: \"kubernetes.io/projected/29d176e8-6196-4867-a79b-b78b51101671-kube-api-access-954vk\") pod \"payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.302830 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.302808 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6"] Sep 17 17:45:29.403109 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.403069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954vk\" (UniqueName: \"kubernetes.io/projected/29d176e8-6196-4867-a79b-b78b51101671-kube-api-access-954vk\") pod \"payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.403253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.403185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d176e8-6196-4867-a79b-b78b51101671-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.403901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.403880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d176e8-6196-4867-a79b-b78b51101671-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.412422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.412396 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-954vk\" (UniqueName: \"kubernetes.io/projected/29d176e8-6196-4867-a79b-b78b51101671-kube-api-access-954vk\") pod \"payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.603775 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.603735 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:29.881487 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:29.881463 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6"] Sep 17 17:45:29.882527 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:29.882496 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d176e8_6196_4867_a79b_b78b51101671.slice/crio-8b4f5a32c4c10ba13622f307e319d14a767fa837d78a0127c1f1de9d0edefc3a WatchSource:0}: Error finding container 8b4f5a32c4c10ba13622f307e319d14a767fa837d78a0127c1f1de9d0edefc3a: Status 404 returned error can't find the container with id 8b4f5a32c4c10ba13622f307e319d14a767fa837d78a0127c1f1de9d0edefc3a Sep 17 17:45:30.629366 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.629332 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2"] Sep 17 17:45:30.633498 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.633476 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.636079 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.636053 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dfb277-serving-cert\"" Sep 17 17:45:30.644270 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.644247 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2"] Sep 17 17:45:30.705708 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.705665 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" event={"ID":"29d176e8-6196-4867-a79b-b78b51101671","Type":"ContainerStarted","Data":"ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9"} Sep 17 17:45:30.705877 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.705749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" event={"ID":"29d176e8-6196-4867-a79b-b78b51101671","Type":"ContainerStarted","Data":"8b4f5a32c4c10ba13622f307e319d14a767fa837d78a0127c1f1de9d0edefc3a"} Sep 17 17:45:30.705985 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.705886 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:30.715010 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.714982 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-metrics-certs\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.715010 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.715022 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg7rm\" (UniqueName: \"kubernetes.io/projected/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-kube-api-access-dg7rm\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.715210 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.715044 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.715210 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.715113 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.726344 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.726297 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" podStartSLOduration=1.726280398 podStartE2EDuration="1.726280398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:30.724180217 +0000 UTC m=+867.286345076" watchObservedRunningTime="2026-09-17 17:45:30.726280398 +0000 UTC m=+867.288445258" Sep 17 17:45:30.816554 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.816512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.816754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.816579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-metrics-certs\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.816754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.816607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7rm\" (UniqueName: \"kubernetes.io/projected/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-kube-api-access-dg7rm\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.816754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.816628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.816754 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:30.816742 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-serving-cert: secret "maas-api-e2e-inf-b-dfb277-serving-cert" not found Sep 17 17:45:30.816948 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:30.816810 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls podName:ef15b92a-bda7-48a1-a2dd-029e3bb847f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:31.316789032 +0000 UTC m=+867.878953873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls") pod "maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" (UID: "ef15b92a-bda7-48a1-a2dd-029e3bb847f7") : secret "maas-api-e2e-inf-b-dfb277-serving-cert" not found Sep 17 17:45:30.817335 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.817314 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.819262 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.819236 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-metrics-certs\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:30.828288 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:30.828264 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg7rm\" (UniqueName: \"kubernetes.io/projected/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-kube-api-access-dg7rm\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:31.320813 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:31.320777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:31.321001 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:31.320923 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-serving-cert: secret "maas-api-e2e-inf-b-dfb277-serving-cert" not found Sep 17 17:45:31.321067 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:31.321054 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls podName:ef15b92a-bda7-48a1-a2dd-029e3bb847f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:32.321036473 +0000 UTC m=+868.883201312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls") pod "maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" (UID: "ef15b92a-bda7-48a1-a2dd-029e3bb847f7") : secret "maas-api-e2e-inf-b-dfb277-serving-cert" not found Sep 17 17:45:32.328980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:32.328926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:32.331321 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:32.331294 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:32.446014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:32.445978 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:32.665549 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:32.665445 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:32.665740 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:32.665609 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:32.723518 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:32.723488 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2"] Sep 17 17:45:32.725157 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:32.725126 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef15b92a_bda7_48a1_a2dd_029e3bb847f7.slice/crio-4ba6b9ddf94cb7d158f85dd50da3ae8f6dcaffc532a3c0ba8499d6f182625ca8 WatchSource:0}: Error finding container 4ba6b9ddf94cb7d158f85dd50da3ae8f6dcaffc532a3c0ba8499d6f182625ca8: Status 404 returned error can't find the container with id 4ba6b9ddf94cb7d158f85dd50da3ae8f6dcaffc532a3c0ba8499d6f182625ca8 Sep 17 17:45:33.723460 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:33.723428 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" event={"ID":"ef15b92a-bda7-48a1-a2dd-029e3bb847f7","Type":"ContainerStarted","Data":"d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b"} Sep 17 17:45:33.723839 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:33.723486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" event={"ID":"ef15b92a-bda7-48a1-a2dd-029e3bb847f7","Type":"ContainerStarted","Data":"4ba6b9ddf94cb7d158f85dd50da3ae8f6dcaffc532a3c0ba8499d6f182625ca8"} Sep 17 17:45:33.723839 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:33.723551 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:39.734617 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:39.734588 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:45:39.770725 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:39.770621 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" podStartSLOduration=9.464536257 podStartE2EDuration="9.770602399s" totalImagesPullingTime="306.066142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:33.755322878 +0000 UTC m=+870.317487738" watchObservedRunningTime="2026-09-17 17:45:39.770602399 +0000 UTC m=+876.332767261" Sep 17 17:45:41.712564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:41.712532 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:45:45.173578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.173546 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h"] Sep 17 17:45:45.174161 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.174109 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" podUID="ac32abfe-5c80-494e-b45e-1cb820531201" containerName="maas-api" containerID="cri-o://afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144" gracePeriod=30 Sep 17 17:45:45.239614 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.239579 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5"] Sep 17 17:45:45.239840 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.239814 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" podUID="f0f1fe87-393e-40cc-b509-a9697a6b3a32" containerName="payload-processing" containerID="cri-o://c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6" gracePeriod=30 Sep 17 17:45:45.249175 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.249005 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr"] Sep 17 17:45:45.249479 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.249445 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" podUID="b5802f0e-0942-4496-91fd-f99d68e370aa" containerName="payload-pre-processing" containerID="cri-o://8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a" gracePeriod=30 Sep 17 17:45:45.534386 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.534357 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:45.537303 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.537279 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:45.540840 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.540820 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:45.645252 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645218 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac32abfe-5c80-494e-b45e-1cb820531201-openshift-service-ca\") pod \"ac32abfe-5c80-494e-b45e-1cb820531201\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " Sep 17 17:45:45.645426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645263 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8t2\" (UniqueName: \"kubernetes.io/projected/f0f1fe87-393e-40cc-b509-a9697a6b3a32-kube-api-access-lz8t2\") pod \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " Sep 17 17:45:45.645426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645294 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-metrics-certs\") pod \"ac32abfe-5c80-494e-b45e-1cb820531201\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " Sep 17 17:45:45.645426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645325 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f1fe87-393e-40cc-b509-a9697a6b3a32-plugins-config-volume\") pod \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\" (UID: \"f0f1fe87-393e-40cc-b509-a9697a6b3a32\") " Sep 17 17:45:45.645426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94m7c\" (UniqueName: \"kubernetes.io/projected/ac32abfe-5c80-494e-b45e-1cb820531201-kube-api-access-94m7c\") pod \"ac32abfe-5c80-494e-b45e-1cb820531201\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " Sep 17 17:45:45.645729 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645447 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls\") pod \"ac32abfe-5c80-494e-b45e-1cb820531201\" (UID: \"ac32abfe-5c80-494e-b45e-1cb820531201\") " Sep 17 17:45:45.645729 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645472 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5802f0e-0942-4496-91fd-f99d68e370aa-plugins-config-volume\") pod \"b5802f0e-0942-4496-91fd-f99d68e370aa\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " Sep 17 17:45:45.645729 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645508 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rj2j\" (UniqueName: \"kubernetes.io/projected/b5802f0e-0942-4496-91fd-f99d68e370aa-kube-api-access-5rj2j\") pod \"b5802f0e-0942-4496-91fd-f99d68e370aa\" (UID: \"b5802f0e-0942-4496-91fd-f99d68e370aa\") " Sep 17 17:45:45.645729 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645615 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac32abfe-5c80-494e-b45e-1cb820531201-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac32abfe-5c80-494e-b45e-1cb820531201" (UID: "ac32abfe-5c80-494e-b45e-1cb820531201"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:45:45.645922 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.645776 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac32abfe-5c80-494e-b45e-1cb820531201-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.647350 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.647128 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f1fe87-393e-40cc-b509-a9697a6b3a32-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0f1fe87-393e-40cc-b509-a9697a6b3a32" (UID: "f0f1fe87-393e-40cc-b509-a9697a6b3a32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:45:45.650498 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.648383 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5802f0e-0942-4496-91fd-f99d68e370aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5802f0e-0942-4496-91fd-f99d68e370aa" (UID: "b5802f0e-0942-4496-91fd-f99d68e370aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:45:45.660520 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.660485 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac32abfe-5c80-494e-b45e-1cb820531201-kube-api-access-94m7c" (OuterVolumeSpecName: "kube-api-access-94m7c") pod "ac32abfe-5c80-494e-b45e-1cb820531201" (UID: "ac32abfe-5c80-494e-b45e-1cb820531201"). InnerVolumeSpecName "kube-api-access-94m7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:45:45.660917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.660889 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f1fe87-393e-40cc-b509-a9697a6b3a32-kube-api-access-lz8t2" (OuterVolumeSpecName: "kube-api-access-lz8t2") pod "f0f1fe87-393e-40cc-b509-a9697a6b3a32" (UID: "f0f1fe87-393e-40cc-b509-a9697a6b3a32"). InnerVolumeSpecName "kube-api-access-lz8t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:45:45.662097 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.662061 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac32abfe-5c80-494e-b45e-1cb820531201" (UID: "ac32abfe-5c80-494e-b45e-1cb820531201"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:45:45.663439 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.663405 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5802f0e-0942-4496-91fd-f99d68e370aa-kube-api-access-5rj2j" (OuterVolumeSpecName: "kube-api-access-5rj2j") pod "b5802f0e-0942-4496-91fd-f99d68e370aa" (UID: "b5802f0e-0942-4496-91fd-f99d68e370aa"). InnerVolumeSpecName "kube-api-access-5rj2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:45:45.664934 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.664909 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac32abfe-5c80-494e-b45e-1cb820531201" (UID: "ac32abfe-5c80-494e-b45e-1cb820531201"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746472 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94m7c\" (UniqueName: \"kubernetes.io/projected/ac32abfe-5c80-494e-b45e-1cb820531201-kube-api-access-94m7c\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746506 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746516 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5802f0e-0942-4496-91fd-f99d68e370aa-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746524 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rj2j\" (UniqueName: \"kubernetes.io/projected/b5802f0e-0942-4496-91fd-f99d68e370aa-kube-api-access-5rj2j\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746532 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8t2\" (UniqueName: \"kubernetes.io/projected/f0f1fe87-393e-40cc-b509-a9697a6b3a32-kube-api-access-lz8t2\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746544 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac32abfe-5c80-494e-b45e-1cb820531201-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.746557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.746556 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f1fe87-393e-40cc-b509-a9697a6b3a32-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:45:45.787385 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.787347 2566 generic.go:356] "Generic (PLEG): container finished" podID="f0f1fe87-393e-40cc-b509-a9697a6b3a32" containerID="c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6" exitCode=0 Sep 17 17:45:45.787385 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.787378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" event={"ID":"f0f1fe87-393e-40cc-b509-a9697a6b3a32","Type":"ContainerDied","Data":"c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6"} Sep 17 17:45:45.787649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.787420 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" Sep 17 17:45:45.787649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.787466 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5" event={"ID":"f0f1fe87-393e-40cc-b509-a9697a6b3a32","Type":"ContainerDied","Data":"ab855bfa0dea7e1fc627691fc685553bb488714b79667e3541ac7b114cf7a88a"} Sep 17 17:45:45.787649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.787506 2566 scope.go:117] "RemoveContainer" containerID="c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6" Sep 17 17:45:45.789450 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.789419 2566 generic.go:356] "Generic (PLEG): container finished" podID="b5802f0e-0942-4496-91fd-f99d68e370aa" containerID="8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a" exitCode=0 Sep 17 17:45:45.789450 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.789454 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" Sep 17 17:45:45.789636 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.789507 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" event={"ID":"b5802f0e-0942-4496-91fd-f99d68e370aa","Type":"ContainerDied","Data":"8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a"} Sep 17 17:45:45.789636 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.789581 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr" event={"ID":"b5802f0e-0942-4496-91fd-f99d68e370aa","Type":"ContainerDied","Data":"1f4f4b53943887acad7398bce2c358f4a66417281002c27ebbcb74688e2b8187"} Sep 17 17:45:45.791331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.791307 2566 generic.go:356] "Generic (PLEG): container finished" podID="ac32abfe-5c80-494e-b45e-1cb820531201" containerID="afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144" exitCode=0 Sep 17 17:45:45.791456 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.791342 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" event={"ID":"ac32abfe-5c80-494e-b45e-1cb820531201","Type":"ContainerDied","Data":"afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144"} Sep 17 17:45:45.791456 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.791379 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" event={"ID":"ac32abfe-5c80-494e-b45e-1cb820531201","Type":"ContainerDied","Data":"6987f55d53c2e66bbf6fcfe74f509de7fc8873a293ef9820d3f0cfb341ea3ca6"} Sep 17 17:45:45.791698 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.791678 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h" Sep 17 17:45:45.797739 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.797723 2566 scope.go:117] "RemoveContainer" containerID="c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6" Sep 17 17:45:45.798045 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:45.798017 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6\": container with ID starting with c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6 not found: ID does not exist" containerID="c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6" Sep 17 17:45:45.798143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.798055 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6"} err="failed to get container status \"c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6\": rpc error: code = NotFound desc = could not find container \"c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6\": container with ID starting with c1eac410ffe3f7228b5ee9590c87573b5eb0c5233bab8939ce1be79a352846f6 not found: ID does not exist" Sep 17 17:45:45.798143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.798077 2566 scope.go:117] "RemoveContainer" containerID="8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a" Sep 17 17:45:45.806333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.806315 2566 scope.go:117] "RemoveContainer" containerID="8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a" Sep 17 17:45:45.806597 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:45.806578 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a\": container with ID starting with 8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a not found: ID does not exist" containerID="8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a" Sep 17 17:45:45.806643 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.806604 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a"} err="failed to get container status \"8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a\": rpc error: code = NotFound desc = could not find container \"8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a\": container with ID starting with 8926f8c6739b4094a03b33b3306fbad9425ed1f10fa79be1a7e77d87535cc92a not found: ID does not exist" Sep 17 17:45:45.806643 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.806619 2566 scope.go:117] "RemoveContainer" containerID="afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144" Sep 17 17:45:45.813128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.813108 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5"] Sep 17 17:45:45.814431 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.814412 2566 scope.go:117] "RemoveContainer" containerID="afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144" Sep 17 17:45:45.814671 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:45.814651 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144\": container with ID starting with afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144 not found: ID does not exist" containerID="afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144" Sep 17 17:45:45.814743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.814675 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144"} err="failed to get container status \"afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144\": rpc error: code = NotFound desc = could not find container \"afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144\": container with ID starting with afe15c6950e50912a9039546ee4eddb9ce43d4d77e35cd28aba626a8ab72d144 not found: ID does not exist" Sep 17 17:45:45.817341 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.817322 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7d735f2-79f74945dc-2xjk5"] Sep 17 17:45:45.829288 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.829270 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h"] Sep 17 17:45:45.834359 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.834340 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7d735f2-545c54bb76-sc79h"] Sep 17 17:45:45.846088 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.846066 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr"] Sep 17 17:45:45.849898 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:45.849872 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7d735f2-cb7794cd-wgpzr"] Sep 17 17:45:46.149857 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:46.149826 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac32abfe-5c80-494e-b45e-1cb820531201" path="/var/lib/kubelet/pods/ac32abfe-5c80-494e-b45e-1cb820531201/volumes" Sep 17 17:45:46.150225 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:46.150211 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5802f0e-0942-4496-91fd-f99d68e370aa" path="/var/lib/kubelet/pods/b5802f0e-0942-4496-91fd-f99d68e370aa/volumes" Sep 17 17:45:46.150536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:46.150525 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0f1fe87-393e-40cc-b509-a9697a6b3a32" path="/var/lib/kubelet/pods/f0f1fe87-393e-40cc-b509-a9697a6b3a32/volumes" Sep 17 17:45:52.711124 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711088 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s"] Sep 17 17:45:52.711655 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711637 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f1fe87-393e-40cc-b509-a9697a6b3a32" containerName="payload-processing" Sep 17 17:45:52.711741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711658 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f1fe87-393e-40cc-b509-a9697a6b3a32" containerName="payload-processing" Sep 17 17:45:52.711741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711675 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac32abfe-5c80-494e-b45e-1cb820531201" containerName="maas-api" Sep 17 17:45:52.711741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711684 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac32abfe-5c80-494e-b45e-1cb820531201" containerName="maas-api" Sep 17 17:45:52.711741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711699 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5802f0e-0942-4496-91fd-f99d68e370aa" containerName="payload-pre-processing" Sep 17 17:45:52.711741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711709 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5802f0e-0942-4496-91fd-f99d68e370aa" containerName="payload-pre-processing" Sep 17 17:45:52.712005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711795 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f1fe87-393e-40cc-b509-a9697a6b3a32" containerName="payload-processing" Sep 17 17:45:52.712005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711810 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac32abfe-5c80-494e-b45e-1cb820531201" containerName="maas-api" Sep 17 17:45:52.712005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.711821 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5802f0e-0942-4496-91fd-f99d68e370aa" containerName="payload-pre-processing" Sep 17 17:45:52.716128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.716106 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.719170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.719150 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-49bc2f4e-serving-cert\"" Sep 17 17:45:52.733724 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.733700 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s"] Sep 17 17:45:52.813517 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.813492 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6wp\" (UniqueName: \"kubernetes.io/projected/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-kube-api-access-nh6wp\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.813618 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.813536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-metrics-certs\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.813618 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.813567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-openshift-service-ca\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.813618 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.813597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.914069 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.914030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6wp\" (UniqueName: \"kubernetes.io/projected/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-kube-api-access-nh6wp\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.914291 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.914103 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-metrics-certs\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.914291 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.914156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-openshift-service-ca\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.914291 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.914188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.914449 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:52.914307 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-serving-cert: secret "maas-api-e2e-migrate-49bc2f4e-serving-cert" not found Sep 17 17:45:52.914449 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:45:52.914431 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls podName:1c5840f3-06b7-4263-be9a-2f11c46bb2cb nodeName:}" failed. No retries permitted until 2026-09-17 17:45:53.414409972 +0000 UTC m=+889.976574833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls") pod "maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" (UID: "1c5840f3-06b7-4263-be9a-2f11c46bb2cb") : secret "maas-api-e2e-migrate-49bc2f4e-serving-cert" not found Sep 17 17:45:52.915073 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.915053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-openshift-service-ca\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.916632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.916610 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-metrics-certs\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:52.930264 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:52.930236 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6wp\" (UniqueName: \"kubernetes.io/projected/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-kube-api-access-nh6wp\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:53.418803 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:53.418767 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:53.421152 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:53.421129 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls\") pod \"maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:53.646079 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:53.646047 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:53.965446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:53.965422 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s"] Sep 17 17:45:53.967880 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:53.967839 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5840f3_06b7_4263_be9a_2f11c46bb2cb.slice/crio-21589b31e23a33660ae7f0048b1eb8b495b2a21b0b7f2465eeb0929ee11b42fb WatchSource:0}: Error finding container 21589b31e23a33660ae7f0048b1eb8b495b2a21b0b7f2465eeb0929ee11b42fb: Status 404 returned error can't find the container with id 21589b31e23a33660ae7f0048b1eb8b495b2a21b0b7f2465eeb0929ee11b42fb Sep 17 17:45:54.222190 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.222105 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b"] Sep 17 17:45:54.226786 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.226766 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.231776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.231756 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-49bc2f4e-dockercfg-twrg9\"" Sep 17 17:45:54.233754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.233640 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-49bc2f4e\"" Sep 17 17:45:54.247457 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.247435 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b"] Sep 17 17:45:54.325389 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.325360 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.325536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.325420 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7b6k\" (UniqueName: \"kubernetes.io/projected/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-kube-api-access-f7b6k\") pod \"payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.426619 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.426577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.426776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.426672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7b6k\" (UniqueName: \"kubernetes.io/projected/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-kube-api-access-f7b6k\") pod \"payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.427271 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.427250 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.443810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.443780 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7b6k\" (UniqueName: \"kubernetes.io/projected/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-kube-api-access-f7b6k\") pod \"payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.449370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.449346 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng"] Sep 17 17:45:54.452965 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.452936 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.468900 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.468868 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng"] Sep 17 17:45:54.527885 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.527809 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjth\" (UniqueName: \"kubernetes.io/projected/4526a716-61be-47f8-a960-5d9f05cf89dc-kube-api-access-smjth\") pod \"payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.527885 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.527867 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526a716-61be-47f8-a960-5d9f05cf89dc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.537084 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.537061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:45:54.628547 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.628516 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526a716-61be-47f8-a960-5d9f05cf89dc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.628688 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.628593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjth\" (UniqueName: \"kubernetes.io/projected/4526a716-61be-47f8-a960-5d9f05cf89dc-kube-api-access-smjth\") pod \"payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.629305 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.629285 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526a716-61be-47f8-a960-5d9f05cf89dc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.640388 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.640366 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjth\" (UniqueName: \"kubernetes.io/projected/4526a716-61be-47f8-a960-5d9f05cf89dc-kube-api-access-smjth\") pod \"payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.762822 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.762791 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:54.816939 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.816905 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b"] Sep 17 17:45:54.819764 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:54.819630 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8adbf9_8fab_4be0_a3ab_4a2be22c24d0.slice/crio-7fb63a6c1e1a0a61387a30742d9f6ddb348377378647516aad49da5381cea2f9 WatchSource:0}: Error finding container 7fb63a6c1e1a0a61387a30742d9f6ddb348377378647516aad49da5381cea2f9: Status 404 returned error can't find the container with id 7fb63a6c1e1a0a61387a30742d9f6ddb348377378647516aad49da5381cea2f9 Sep 17 17:45:54.839563 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.839532 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" event={"ID":"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0","Type":"ContainerStarted","Data":"7fb63a6c1e1a0a61387a30742d9f6ddb348377378647516aad49da5381cea2f9"} Sep 17 17:45:54.841515 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.841471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" event={"ID":"1c5840f3-06b7-4263-be9a-2f11c46bb2cb","Type":"ContainerStarted","Data":"98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6"} Sep 17 17:45:54.841515 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.841520 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" event={"ID":"1c5840f3-06b7-4263-be9a-2f11c46bb2cb","Type":"ContainerStarted","Data":"21589b31e23a33660ae7f0048b1eb8b495b2a21b0b7f2465eeb0929ee11b42fb"} Sep 17 17:45:54.841701 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.841599 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:45:54.863226 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:54.863184 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" podStartSLOduration=2.501071229 podStartE2EDuration="2.863167128s" totalImagesPullingTime="362.095899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:54.861933307 +0000 UTC m=+891.424098168" watchObservedRunningTime="2026-09-17 17:45:54.863167128 +0000 UTC m=+891.425331988" Sep 17 17:45:55.042906 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.042874 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng"] Sep 17 17:45:55.043395 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:45:55.043372 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4526a716_61be_47f8_a960_5d9f05cf89dc.slice/crio-0c0aa98d17838cc4274de0ed29ba8077569a8ca4f21b51e59ee8296e6274179a WatchSource:0}: Error finding container 0c0aa98d17838cc4274de0ed29ba8077569a8ca4f21b51e59ee8296e6274179a: Status 404 returned error can't find the container with id 0c0aa98d17838cc4274de0ed29ba8077569a8ca4f21b51e59ee8296e6274179a Sep 17 17:45:55.850925 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.850878 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" event={"ID":"4526a716-61be-47f8-a960-5d9f05cf89dc","Type":"ContainerStarted","Data":"16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1"} Sep 17 17:45:55.851141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.850950 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" event={"ID":"4526a716-61be-47f8-a960-5d9f05cf89dc","Type":"ContainerStarted","Data":"0c0aa98d17838cc4274de0ed29ba8077569a8ca4f21b51e59ee8296e6274179a"} Sep 17 17:45:55.851141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.850994 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:45:55.852469 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.852439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" event={"ID":"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0","Type":"ContainerStarted","Data":"addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49"} Sep 17 17:45:55.872168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.872122 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" podStartSLOduration=1.872106868 podStartE2EDuration="1.872106868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:55.870811831 +0000 UTC m=+892.432976685" watchObservedRunningTime="2026-09-17 17:45:55.872106868 +0000 UTC m=+892.434271727" Sep 17 17:45:55.891495 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:55.891424 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" podStartSLOduration=1.891405651 podStartE2EDuration="1.891405651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:55.88935991 +0000 UTC m=+892.451524782" watchObservedRunningTime="2026-09-17 17:45:55.891405651 +0000 UTC m=+892.453570512" Sep 17 17:45:56.857533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:45:56.857456 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:46:00.857808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:00.857777 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:46:06.859298 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:06.859266 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:46:07.864474 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:07.864447 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:46:15.284158 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.284119 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s"] Sep 17 17:46:15.284690 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.284383 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" podUID="1c5840f3-06b7-4263-be9a-2f11c46bb2cb" containerName="maas-api" containerID="cri-o://98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6" gracePeriod=30 Sep 17 17:46:15.351290 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.351256 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng"] Sep 17 17:46:15.351580 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.351549 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" podUID="4526a716-61be-47f8-a960-5d9f05cf89dc" containerName="payload-processing" containerID="cri-o://16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1" gracePeriod=30 Sep 17 17:46:15.359051 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.359022 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b"] Sep 17 17:46:15.359603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.359556 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" podUID="ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" containerName="payload-pre-processing" containerID="cri-o://addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49" gracePeriod=30 Sep 17 17:46:15.631376 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.631349 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:46:15.648192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.648172 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:46:15.655452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.655425 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:46:15.723858 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.723819 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls\") pod \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " Sep 17 17:46:15.723858 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.723854 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526a716-61be-47f8-a960-5d9f05cf89dc-plugins-config-volume\") pod \"4526a716-61be-47f8-a960-5d9f05cf89dc\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " Sep 17 17:46:15.724141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.723875 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh6wp\" (UniqueName: \"kubernetes.io/projected/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-kube-api-access-nh6wp\") pod \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " Sep 17 17:46:15.724141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.723889 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smjth\" (UniqueName: \"kubernetes.io/projected/4526a716-61be-47f8-a960-5d9f05cf89dc-kube-api-access-smjth\") pod \"4526a716-61be-47f8-a960-5d9f05cf89dc\" (UID: \"4526a716-61be-47f8-a960-5d9f05cf89dc\") " Sep 17 17:46:15.724141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.723914 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-metrics-certs\") pod \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " Sep 17 17:46:15.724141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.723949 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7b6k\" (UniqueName: \"kubernetes.io/projected/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-kube-api-access-f7b6k\") pod \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " Sep 17 17:46:15.724141 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.724052 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-openshift-service-ca\") pod \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\" (UID: \"1c5840f3-06b7-4263-be9a-2f11c46bb2cb\") " Sep 17 17:46:15.724422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.724322 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526a716-61be-47f8-a960-5d9f05cf89dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4526a716-61be-47f8-a960-5d9f05cf89dc" (UID: "4526a716-61be-47f8-a960-5d9f05cf89dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:46:15.724701 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.724673 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c5840f3-06b7-4263-be9a-2f11c46bb2cb" (UID: "1c5840f3-06b7-4263-be9a-2f11c46bb2cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:46:15.725245 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.724684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-plugins-config-volume\") pod \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\" (UID: \"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0\") " Sep 17 17:46:15.725245 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.725074 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" (UID: "ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:46:15.725409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.725339 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.725409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.725361 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.725409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.725376 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4526a716-61be-47f8-a960-5d9f05cf89dc-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.726828 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.726614 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4526a716-61be-47f8-a960-5d9f05cf89dc-kube-api-access-smjth" (OuterVolumeSpecName: "kube-api-access-smjth") pod "4526a716-61be-47f8-a960-5d9f05cf89dc" (UID: "4526a716-61be-47f8-a960-5d9f05cf89dc"). InnerVolumeSpecName "kube-api-access-smjth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:46:15.726828 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.726691 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c5840f3-06b7-4263-be9a-2f11c46bb2cb" (UID: "1c5840f3-06b7-4263-be9a-2f11c46bb2cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:46:15.726828 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.726727 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-kube-api-access-nh6wp" (OuterVolumeSpecName: "kube-api-access-nh6wp") pod "1c5840f3-06b7-4263-be9a-2f11c46bb2cb" (UID: "1c5840f3-06b7-4263-be9a-2f11c46bb2cb"). InnerVolumeSpecName "kube-api-access-nh6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:46:15.726828 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.726790 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c5840f3-06b7-4263-be9a-2f11c46bb2cb" (UID: "1c5840f3-06b7-4263-be9a-2f11c46bb2cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:46:15.727227 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.727198 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-kube-api-access-f7b6k" (OuterVolumeSpecName: "kube-api-access-f7b6k") pod "ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" (UID: "ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0"). InnerVolumeSpecName "kube-api-access-f7b6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:46:15.826029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.825929 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.826029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.825978 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7b6k\" (UniqueName: \"kubernetes.io/projected/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0-kube-api-access-f7b6k\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.826029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.825994 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.826029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.826004 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6wp\" (UniqueName: \"kubernetes.io/projected/1c5840f3-06b7-4263-be9a-2f11c46bb2cb-kube-api-access-nh6wp\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.826029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.826012 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smjth\" (UniqueName: \"kubernetes.io/projected/4526a716-61be-47f8-a960-5d9f05cf89dc-kube-api-access-smjth\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:15.964000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.963948 2566 generic.go:356] "Generic (PLEG): container finished" podID="ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" containerID="addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49" exitCode=0 Sep 17 17:46:15.964168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.964019 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" Sep 17 17:46:15.964168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.964044 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" event={"ID":"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0","Type":"ContainerDied","Data":"addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49"} Sep 17 17:46:15.964168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.964099 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b" event={"ID":"ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0","Type":"ContainerDied","Data":"7fb63a6c1e1a0a61387a30742d9f6ddb348377378647516aad49da5381cea2f9"} Sep 17 17:46:15.964168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.964128 2566 scope.go:117] "RemoveContainer" containerID="addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49" Sep 17 17:46:15.965340 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.965316 2566 generic.go:356] "Generic (PLEG): container finished" podID="4526a716-61be-47f8-a960-5d9f05cf89dc" containerID="16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1" exitCode=0 Sep 17 17:46:15.965414 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.965351 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" Sep 17 17:46:15.965414 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.965369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" event={"ID":"4526a716-61be-47f8-a960-5d9f05cf89dc","Type":"ContainerDied","Data":"16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1"} Sep 17 17:46:15.965501 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.965413 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng" event={"ID":"4526a716-61be-47f8-a960-5d9f05cf89dc","Type":"ContainerDied","Data":"0c0aa98d17838cc4274de0ed29ba8077569a8ca4f21b51e59ee8296e6274179a"} Sep 17 17:46:15.967146 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.966909 2566 generic.go:356] "Generic (PLEG): container finished" podID="1c5840f3-06b7-4263-be9a-2f11c46bb2cb" containerID="98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6" exitCode=0 Sep 17 17:46:15.967146 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.967008 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" event={"ID":"1c5840f3-06b7-4263-be9a-2f11c46bb2cb","Type":"ContainerDied","Data":"98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6"} Sep 17 17:46:15.967146 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.967045 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" event={"ID":"1c5840f3-06b7-4263-be9a-2f11c46bb2cb","Type":"ContainerDied","Data":"21589b31e23a33660ae7f0048b1eb8b495b2a21b0b7f2465eeb0929ee11b42fb"} Sep 17 17:46:15.967380 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.967240 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s" Sep 17 17:46:15.980762 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.980743 2566 scope.go:117] "RemoveContainer" containerID="addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49" Sep 17 17:46:15.981088 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:15.981068 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49\": container with ID starting with addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49 not found: ID does not exist" containerID="addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49" Sep 17 17:46:15.981196 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.981095 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49"} err="failed to get container status \"addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49\": rpc error: code = NotFound desc = could not find container \"addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49\": container with ID starting with addd82e2d3e010d9601b5031286ceba8322955bd9201e733b7dffe60e7c38f49 not found: ID does not exist" Sep 17 17:46:15.981196 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.981112 2566 scope.go:117] "RemoveContainer" containerID="16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1" Sep 17 17:46:15.998814 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.998777 2566 scope.go:117] "RemoveContainer" containerID="16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1" Sep 17 17:46:15.999101 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:15.999079 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1\": container with ID starting with 16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1 not found: ID does not exist" containerID="16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1" Sep 17 17:46:15.999171 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.999108 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1"} err="failed to get container status \"16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1\": rpc error: code = NotFound desc = could not find container \"16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1\": container with ID starting with 16964af0ab2944cf94ac3e5c5f20c15739b96fb7cbdf3b8c716e7abfe743c5b1 not found: ID does not exist" Sep 17 17:46:15.999171 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:15.999123 2566 scope.go:117] "RemoveContainer" containerID="98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6" Sep 17 17:46:16.007330 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.007309 2566 scope.go:117] "RemoveContainer" containerID="98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6" Sep 17 17:46:16.007612 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:16.007595 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6\": container with ID starting with 98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6 not found: ID does not exist" containerID="98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6" Sep 17 17:46:16.007673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.007621 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6"} err="failed to get container status \"98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6\": rpc error: code = NotFound desc = could not find container \"98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6\": container with ID starting with 98317d894e47f6481d7d508a3768624a10bf16f7434884a5422406a3e51f6db6 not found: ID does not exist" Sep 17 17:46:16.012399 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.012375 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng"] Sep 17 17:46:16.024902 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.024876 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49bc2f4e-56659b5d94-4nkng"] Sep 17 17:46:16.042665 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.042633 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b"] Sep 17 17:46:16.047458 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.047437 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49bc2f4e-566d687898-j9k7b"] Sep 17 17:46:16.060103 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.060083 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s"] Sep 17 17:46:16.063981 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.063944 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49bc2f4e-76b5dbdd78-8jk8s"] Sep 17 17:46:16.150418 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.150334 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c5840f3-06b7-4263-be9a-2f11c46bb2cb" path="/var/lib/kubelet/pods/1c5840f3-06b7-4263-be9a-2f11c46bb2cb/volumes" Sep 17 17:46:16.150921 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.150901 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4526a716-61be-47f8-a960-5d9f05cf89dc" path="/var/lib/kubelet/pods/4526a716-61be-47f8-a960-5d9f05cf89dc/volumes" Sep 17 17:46:16.151270 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:16.151257 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" path="/var/lib/kubelet/pods/ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0/volumes" Sep 17 17:46:22.303162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303132 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs"] Sep 17 17:46:22.303531 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303508 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" containerName="payload-pre-processing" Sep 17 17:46:22.303531 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303520 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" containerName="payload-pre-processing" Sep 17 17:46:22.303613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303534 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5840f3-06b7-4263-be9a-2f11c46bb2cb" containerName="maas-api" Sep 17 17:46:22.303613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303541 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5840f3-06b7-4263-be9a-2f11c46bb2cb" containerName="maas-api" Sep 17 17:46:22.303613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303551 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4526a716-61be-47f8-a960-5d9f05cf89dc" containerName="payload-processing" Sep 17 17:46:22.303613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303558 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4526a716-61be-47f8-a960-5d9f05cf89dc" containerName="payload-processing" Sep 17 17:46:22.303735 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303624 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c5840f3-06b7-4263-be9a-2f11c46bb2cb" containerName="maas-api" Sep 17 17:46:22.303735 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303631 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4526a716-61be-47f8-a960-5d9f05cf89dc" containerName="payload-processing" Sep 17 17:46:22.303735 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.303639 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8adbf9-8fab-4be0-a3ab-4a2be22c24d0" containerName="payload-pre-processing" Sep 17 17:46:22.308216 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.308196 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.310880 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.310860 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2b0967aa-serving-cert\"" Sep 17 17:46:22.326894 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.326863 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs"] Sep 17 17:46:22.380570 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.380536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.380756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.380593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69w8\" (UniqueName: \"kubernetes.io/projected/2dd761c3-4adb-4bdb-809f-47cbe3994548-kube-api-access-k69w8\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.380756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.380719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd761c3-4adb-4bdb-809f-47cbe3994548-openshift-service-ca\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.380869 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.380753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-metrics-certs\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.482181 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.482152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd761c3-4adb-4bdb-809f-47cbe3994548-openshift-service-ca\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.482325 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.482186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-metrics-certs\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.482325 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.482232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.482325 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.482264 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69w8\" (UniqueName: \"kubernetes.io/projected/2dd761c3-4adb-4bdb-809f-47cbe3994548-kube-api-access-k69w8\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.482470 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:22.482391 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-serving-cert: secret "maas-api-e2e-ait-2b0967aa-serving-cert" not found Sep 17 17:46:22.482470 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:22.482463 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls podName:2dd761c3-4adb-4bdb-809f-47cbe3994548 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:22.982445394 +0000 UTC m=+919.544610233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls") pod "maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" (UID: "2dd761c3-4adb-4bdb-809f-47cbe3994548") : secret "maas-api-e2e-ait-2b0967aa-serving-cert" not found Sep 17 17:46:22.482790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.482763 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd761c3-4adb-4bdb-809f-47cbe3994548-openshift-service-ca\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.485022 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.484991 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-metrics-certs\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.492540 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.492521 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69w8\" (UniqueName: \"kubernetes.io/projected/2dd761c3-4adb-4bdb-809f-47cbe3994548-kube-api-access-k69w8\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.987964 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.987926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:22.990494 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:22.990471 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls\") pod \"maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:23.192856 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.192821 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz"] Sep 17 17:46:23.196482 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.196466 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.199058 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.199037 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2b0967aa\"" Sep 17 17:46:23.199339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.199319 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2b0967aa-dockercfg-xqs5m\"" Sep 17 17:46:23.215225 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.215199 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz"] Sep 17 17:46:23.218340 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.218320 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:23.245885 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.245823 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq"] Sep 17 17:46:23.250193 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.250172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.263613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.263586 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq"] Sep 17 17:46:23.299986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.291081 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.299986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.291140 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.299986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.291165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4sk\" (UniqueName: \"kubernetes.io/projected/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-kube-api-access-cl4sk\") pod \"payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.299986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.291256 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffjv\" (UniqueName: \"kubernetes.io/projected/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-kube-api-access-vffjv\") pod \"payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.392395 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.392353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.392395 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.392387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4sk\" (UniqueName: \"kubernetes.io/projected/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-kube-api-access-cl4sk\") pod \"payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.392967 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.392445 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffjv\" (UniqueName: \"kubernetes.io/projected/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-kube-api-access-vffjv\") pod \"payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.392967 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.392491 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.393170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.393150 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.393233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.393220 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.401670 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.401644 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffjv\" (UniqueName: \"kubernetes.io/projected/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-kube-api-access-vffjv\") pod \"payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.401819 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.401734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4sk\" (UniqueName: \"kubernetes.io/projected/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-kube-api-access-cl4sk\") pod \"payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.504306 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.504282 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs"] Sep 17 17:46:23.505928 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:46:23.505900 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd761c3_4adb_4bdb_809f_47cbe3994548.slice/crio-27dd38a69aa2bc8d55ee4d28def90c90b594d83307e8835c2b5155cfa8b8decd WatchSource:0}: Error finding container 27dd38a69aa2bc8d55ee4d28def90c90b594d83307e8835c2b5155cfa8b8decd: Status 404 returned error can't find the container with id 27dd38a69aa2bc8d55ee4d28def90c90b594d83307e8835c2b5155cfa8b8decd Sep 17 17:46:23.506694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.506679 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:23.563759 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.563511 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:23.795850 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.795824 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz"] Sep 17 17:46:23.797159 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:46:23.797139 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7bae93_6682_4a4d_bf5b_3dbed82f21e1.slice/crio-54fb32f78df6c82bd8d533e480f0de36627b9b7d43e5242cc2358d8d592b1a36 WatchSource:0}: Error finding container 54fb32f78df6c82bd8d533e480f0de36627b9b7d43e5242cc2358d8d592b1a36: Status 404 returned error can't find the container with id 54fb32f78df6c82bd8d533e480f0de36627b9b7d43e5242cc2358d8d592b1a36 Sep 17 17:46:23.850117 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:23.850091 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq"] Sep 17 17:46:23.851118 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:46:23.851092 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e916cf_ab53_4538_9bef_f9ae912eb5c4.slice/crio-570ec8dab80d83fe87ffd760b75da0f5c98efb1dd2734fd7b0ca6103dbff2c9b WatchSource:0}: Error finding container 570ec8dab80d83fe87ffd760b75da0f5c98efb1dd2734fd7b0ca6103dbff2c9b: Status 404 returned error can't find the container with id 570ec8dab80d83fe87ffd760b75da0f5c98efb1dd2734fd7b0ca6103dbff2c9b Sep 17 17:46:24.018821 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.018788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" event={"ID":"2dd761c3-4adb-4bdb-809f-47cbe3994548","Type":"ContainerStarted","Data":"21ae0e465a133e16fff320f12b2a14d3324db5a154f76723430e2a4908308d13"} Sep 17 17:46:24.019030 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.018849 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" event={"ID":"2dd761c3-4adb-4bdb-809f-47cbe3994548","Type":"ContainerStarted","Data":"27dd38a69aa2bc8d55ee4d28def90c90b594d83307e8835c2b5155cfa8b8decd"} Sep 17 17:46:24.019030 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.018901 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:24.020546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.020524 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" event={"ID":"d3e916cf-ab53-4538-9bef-f9ae912eb5c4","Type":"ContainerStarted","Data":"9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3"} Sep 17 17:46:24.020656 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.020561 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" event={"ID":"d3e916cf-ab53-4538-9bef-f9ae912eb5c4","Type":"ContainerStarted","Data":"570ec8dab80d83fe87ffd760b75da0f5c98efb1dd2734fd7b0ca6103dbff2c9b"} Sep 17 17:46:24.020656 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.020581 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:24.022251 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.022219 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" event={"ID":"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1","Type":"ContainerStarted","Data":"dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250"} Sep 17 17:46:24.022355 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.022261 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" event={"ID":"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1","Type":"ContainerStarted","Data":"54fb32f78df6c82bd8d533e480f0de36627b9b7d43e5242cc2358d8d592b1a36"} Sep 17 17:46:24.022355 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.022296 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:24.041322 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.041216 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" podStartSLOduration=1.6362486189999998 podStartE2EDuration="2.041199595s" totalImagesPullingTime="404.950976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:24.03991493 +0000 UTC m=+920.602079792" watchObservedRunningTime="2026-09-17 17:46:24.041199595 +0000 UTC m=+920.603364456" Sep 17 17:46:24.064361 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.064312 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" podStartSLOduration=1.064300242 podStartE2EDuration="1.064300242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:24.062543789 +0000 UTC m=+920.624708650" watchObservedRunningTime="2026-09-17 17:46:24.064300242 +0000 UTC m=+920.626465102" Sep 17 17:46:24.087490 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:24.087445 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" podStartSLOduration=1.087428217 podStartE2EDuration="1.087428217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:24.086101222 +0000 UTC m=+920.648266082" watchObservedRunningTime="2026-09-17 17:46:24.087428217 +0000 UTC m=+920.649593076" Sep 17 17:46:30.033043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:30.033012 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:35.029706 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:35.029673 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:35.030121 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:35.029766 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:49.641553 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.641519 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79"] Sep 17 17:46:49.645161 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.645141 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.647801 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.647777 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"kube-root-ca.crt\"" Sep 17 17:46:49.648979 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.648939 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"openshift-service-ca.crt\"" Sep 17 17:46:49.648979 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.648946 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"default-dockercfg-8k5x9\"" Sep 17 17:46:49.649120 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.648939 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"simulator-w4-8aeea1-kserve-self-signed-certs\"" Sep 17 17:46:49.660494 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.660469 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79"] Sep 17 17:46:49.720503 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.720474 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-home\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.720665 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.720512 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5btj\" (UniqueName: \"kubernetes.io/projected/159ff45e-9ede-4787-b935-17910da974f7-kube-api-access-m5btj\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.720665 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.720538 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ff45e-9ede-4787-b935-17910da974f7-tls-certs\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.720665 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.720560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-tmp-dir\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.720665 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.720591 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-model-cache\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.720665 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.720632 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-dshm\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ff45e-9ede-4787-b935-17910da974f7-tls-certs\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821168 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-tmp-dir\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821433 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-model-cache\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821433 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821227 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-dshm\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821433 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821288 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-home\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821433 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5btj\" (UniqueName: \"kubernetes.io/projected/159ff45e-9ede-4787-b935-17910da974f7-kube-api-access-m5btj\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-tmp-dir\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821597 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-model-cache\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.821641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.821635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-home\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.823589 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.823562 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-dshm\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.823986 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.823947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ff45e-9ede-4787-b935-17910da974f7-tls-certs\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.829883 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.829857 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5btj\" (UniqueName: \"kubernetes.io/projected/159ff45e-9ede-4787-b935-17910da974f7-kube-api-access-m5btj\") pod \"simulator-w4-8aeea1-kserve-fb769c9b8-f6n79\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:49.955128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:49.955045 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:50.232438 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:50.232358 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79"] Sep 17 17:46:50.238126 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:46:50.238105 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod159ff45e_9ede_4787_b935_17910da974f7.slice/crio-90a207a85e082408bee7e9325be1c0de1a217ea2dee88a9fc6cc7bdb374a4e70 WatchSource:0}: Error finding container 90a207a85e082408bee7e9325be1c0de1a217ea2dee88a9fc6cc7bdb374a4e70: Status 404 returned error can't find the container with id 90a207a85e082408bee7e9325be1c0de1a217ea2dee88a9fc6cc7bdb374a4e70 Sep 17 17:46:51.174700 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:51.174657 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" event={"ID":"159ff45e-9ede-4787-b935-17910da974f7","Type":"ContainerStarted","Data":"70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba"} Sep 17 17:46:51.175161 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:51.174715 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" event={"ID":"159ff45e-9ede-4787-b935-17910da974f7","Type":"ContainerStarted","Data":"90a207a85e082408bee7e9325be1c0de1a217ea2dee88a9fc6cc7bdb374a4e70"} Sep 17 17:46:51.198876 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:51.198827 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" podStartSLOduration=2.198810665 podStartE2EDuration="2.198810665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:51.19825327 +0000 UTC m=+947.760418130" watchObservedRunningTime="2026-09-17 17:46:51.198810665 +0000 UTC m=+947.760975524" Sep 17 17:46:54.016382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.016349 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs"] Sep 17 17:46:54.016808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.016587 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" podUID="2dd761c3-4adb-4bdb-809f-47cbe3994548" containerName="maas-api" containerID="cri-o://21ae0e465a133e16fff320f12b2a14d3324db5a154f76723430e2a4908308d13" gracePeriod=30 Sep 17 17:46:54.086562 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.086529 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq"] Sep 17 17:46:54.086795 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.086769 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" podUID="d3e916cf-ab53-4538-9bef-f9ae912eb5c4" containerName="payload-processing" containerID="cri-o://9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3" gracePeriod=30 Sep 17 17:46:54.097593 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.097541 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz"] Sep 17 17:46:54.097781 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.097758 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" podUID="4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" containerName="payload-pre-processing" containerID="cri-o://dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250" gracePeriod=30 Sep 17 17:46:54.192014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.191972 2566 generic.go:356] "Generic (PLEG): container finished" podID="2dd761c3-4adb-4bdb-809f-47cbe3994548" containerID="21ae0e465a133e16fff320f12b2a14d3324db5a154f76723430e2a4908308d13" exitCode=0 Sep 17 17:46:54.192151 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.192058 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" event={"ID":"2dd761c3-4adb-4bdb-809f-47cbe3994548","Type":"ContainerDied","Data":"21ae0e465a133e16fff320f12b2a14d3324db5a154f76723430e2a4908308d13"} Sep 17 17:46:54.361853 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.361829 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:54.386400 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.386356 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:54.396646 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.396616 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.466402 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69w8\" (UniqueName: \"kubernetes.io/projected/2dd761c3-4adb-4bdb-809f-47cbe3994548-kube-api-access-k69w8\") pod \"2dd761c3-4adb-4bdb-809f-47cbe3994548\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.466486 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls\") pod \"2dd761c3-4adb-4bdb-809f-47cbe3994548\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.466521 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd761c3-4adb-4bdb-809f-47cbe3994548-openshift-service-ca\") pod \"2dd761c3-4adb-4bdb-809f-47cbe3994548\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.466547 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-plugins-config-volume\") pod \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.466633 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vffjv\" (UniqueName: \"kubernetes.io/projected/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-kube-api-access-vffjv\") pod \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\" (UID: \"d3e916cf-ab53-4538-9bef-f9ae912eb5c4\") " Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.466920 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd761c3-4adb-4bdb-809f-47cbe3994548-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2dd761c3-4adb-4bdb-809f-47cbe3994548" (UID: "2dd761c3-4adb-4bdb-809f-47cbe3994548"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.467006 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-metrics-certs\") pod \"2dd761c3-4adb-4bdb-809f-47cbe3994548\" (UID: \"2dd761c3-4adb-4bdb-809f-47cbe3994548\") " Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.467354 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd761c3-4adb-4bdb-809f-47cbe3994548-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.468848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.467004 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3e916cf-ab53-4538-9bef-f9ae912eb5c4" (UID: "d3e916cf-ab53-4538-9bef-f9ae912eb5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:46:54.469975 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.469922 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-kube-api-access-vffjv" (OuterVolumeSpecName: "kube-api-access-vffjv") pod "d3e916cf-ab53-4538-9bef-f9ae912eb5c4" (UID: "d3e916cf-ab53-4538-9bef-f9ae912eb5c4"). InnerVolumeSpecName "kube-api-access-vffjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:46:54.470377 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.470318 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd761c3-4adb-4bdb-809f-47cbe3994548-kube-api-access-k69w8" (OuterVolumeSpecName: "kube-api-access-k69w8") pod "2dd761c3-4adb-4bdb-809f-47cbe3994548" (UID: "2dd761c3-4adb-4bdb-809f-47cbe3994548"). InnerVolumeSpecName "kube-api-access-k69w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:46:54.470510 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.470484 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2dd761c3-4adb-4bdb-809f-47cbe3994548" (UID: "2dd761c3-4adb-4bdb-809f-47cbe3994548"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:46:54.471539 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.471512 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2dd761c3-4adb-4bdb-809f-47cbe3994548" (UID: "2dd761c3-4adb-4bdb-809f-47cbe3994548"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:46:54.568241 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568209 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-plugins-config-volume\") pod \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " Sep 17 17:46:54.568416 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568258 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl4sk\" (UniqueName: \"kubernetes.io/projected/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-kube-api-access-cl4sk\") pod \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\" (UID: \"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1\") " Sep 17 17:46:54.568530 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568514 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.568596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568537 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.568596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568552 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffjv\" (UniqueName: \"kubernetes.io/projected/d3e916cf-ab53-4538-9bef-f9ae912eb5c4-kube-api-access-vffjv\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.568596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568566 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd761c3-4adb-4bdb-809f-47cbe3994548-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.568596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568579 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69w8\" (UniqueName: \"kubernetes.io/projected/2dd761c3-4adb-4bdb-809f-47cbe3994548-kube-api-access-k69w8\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.568817 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.568656 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" (UID: "4c7bae93-6682-4a4d-bf5b-3dbed82f21e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:46:54.570607 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.570576 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-kube-api-access-cl4sk" (OuterVolumeSpecName: "kube-api-access-cl4sk") pod "4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" (UID: "4c7bae93-6682-4a4d-bf5b-3dbed82f21e1"). InnerVolumeSpecName "kube-api-access-cl4sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:46:54.669192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.669158 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl4sk\" (UniqueName: \"kubernetes.io/projected/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-kube-api-access-cl4sk\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:54.669192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:54.669186 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:46:55.197868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.197839 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" Sep 17 17:46:55.198353 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.197865 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs" event={"ID":"2dd761c3-4adb-4bdb-809f-47cbe3994548","Type":"ContainerDied","Data":"27dd38a69aa2bc8d55ee4d28def90c90b594d83307e8835c2b5155cfa8b8decd"} Sep 17 17:46:55.198353 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.197925 2566 scope.go:117] "RemoveContainer" containerID="21ae0e465a133e16fff320f12b2a14d3324db5a154f76723430e2a4908308d13" Sep 17 17:46:55.199382 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.199325 2566 generic.go:356] "Generic (PLEG): container finished" podID="d3e916cf-ab53-4538-9bef-f9ae912eb5c4" containerID="9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3" exitCode=0 Sep 17 17:46:55.199467 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.199367 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" Sep 17 17:46:55.199467 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.199419 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" event={"ID":"d3e916cf-ab53-4538-9bef-f9ae912eb5c4","Type":"ContainerDied","Data":"9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3"} Sep 17 17:46:55.199584 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.199482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq" event={"ID":"d3e916cf-ab53-4538-9bef-f9ae912eb5c4","Type":"ContainerDied","Data":"570ec8dab80d83fe87ffd760b75da0f5c98efb1dd2734fd7b0ca6103dbff2c9b"} Sep 17 17:46:55.202899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.201879 2566 generic.go:356] "Generic (PLEG): container finished" podID="4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" containerID="dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250" exitCode=0 Sep 17 17:46:55.202899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.202178 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" event={"ID":"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1","Type":"ContainerDied","Data":"dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250"} Sep 17 17:46:55.202899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.202253 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" event={"ID":"4c7bae93-6682-4a4d-bf5b-3dbed82f21e1","Type":"ContainerDied","Data":"54fb32f78df6c82bd8d533e480f0de36627b9b7d43e5242cc2358d8d592b1a36"} Sep 17 17:46:55.202899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.202492 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz" Sep 17 17:46:55.215005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.214986 2566 scope.go:117] "RemoveContainer" containerID="9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3" Sep 17 17:46:55.223445 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.223428 2566 scope.go:117] "RemoveContainer" containerID="9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3" Sep 17 17:46:55.223694 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:55.223675 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3\": container with ID starting with 9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3 not found: ID does not exist" containerID="9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3" Sep 17 17:46:55.223751 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.223700 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3"} err="failed to get container status \"9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3\": rpc error: code = NotFound desc = could not find container \"9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3\": container with ID starting with 9acbc55551346cf45ad181a2dcd919ab21fa3af060494385af236e0791a4b7c3 not found: ID does not exist" Sep 17 17:46:55.223751 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.223715 2566 scope.go:117] "RemoveContainer" containerID="dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250" Sep 17 17:46:55.231587 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.231565 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs"] Sep 17 17:46:55.234853 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.234830 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b0967aa-7c9fdcf89b-ssrgs"] Sep 17 17:46:55.236477 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.236461 2566 scope.go:117] "RemoveContainer" containerID="dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250" Sep 17 17:46:55.236744 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:46:55.236722 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250\": container with ID starting with dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250 not found: ID does not exist" containerID="dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250" Sep 17 17:46:55.236813 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.236751 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250"} err="failed to get container status \"dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250\": rpc error: code = NotFound desc = could not find container \"dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250\": container with ID starting with dfe21df8dd1ab23c8935e55baf1516118777e1e6f6b7444a59c20afd3e397250 not found: ID does not exist" Sep 17 17:46:55.246475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.246450 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz"] Sep 17 17:46:55.250929 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.250906 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b0967aa-7ddfcf5dcd-59gpz"] Sep 17 17:46:55.261169 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.261150 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq"] Sep 17 17:46:55.266174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:55.266149 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b0967aa-5656b7cd67-z8pfq"] Sep 17 17:46:56.149948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:56.149918 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd761c3-4adb-4bdb-809f-47cbe3994548" path="/var/lib/kubelet/pods/2dd761c3-4adb-4bdb-809f-47cbe3994548/volumes" Sep 17 17:46:56.150322 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:56.150308 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" path="/var/lib/kubelet/pods/4c7bae93-6682-4a4d-bf5b-3dbed82f21e1/volumes" Sep 17 17:46:56.150617 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:56.150606 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e916cf-ab53-4538-9bef-f9ae912eb5c4" path="/var/lib/kubelet/pods/d3e916cf-ab53-4538-9bef-f9ae912eb5c4/volumes" Sep 17 17:46:59.955360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:59.955326 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:59.955745 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:59.955373 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:46:59.967931 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:46:59.967904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:47:00.241569 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:00.241487 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:47:01.743745 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.743712 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm"] Sep 17 17:47:01.744166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744151 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" containerName="payload-pre-processing" Sep 17 17:47:01.744236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744168 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" containerName="payload-pre-processing" Sep 17 17:47:01.744236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744193 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e916cf-ab53-4538-9bef-f9ae912eb5c4" containerName="payload-processing" Sep 17 17:47:01.744236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744202 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e916cf-ab53-4538-9bef-f9ae912eb5c4" containerName="payload-processing" Sep 17 17:47:01.744236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744213 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd761c3-4adb-4bdb-809f-47cbe3994548" containerName="maas-api" Sep 17 17:47:01.744236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744221 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd761c3-4adb-4bdb-809f-47cbe3994548" containerName="maas-api" Sep 17 17:47:01.744452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744305 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7bae93-6682-4a4d-bf5b-3dbed82f21e1" containerName="payload-pre-processing" Sep 17 17:47:01.744452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744316 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e916cf-ab53-4538-9bef-f9ae912eb5c4" containerName="payload-processing" Sep 17 17:47:01.744452 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.744325 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd761c3-4adb-4bdb-809f-47cbe3994548" containerName="maas-api" Sep 17 17:47:01.747563 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.747542 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.750653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.750633 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9e4e10ed-serving-cert\"" Sep 17 17:47:01.764534 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.764509 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm"] Sep 17 17:47:01.846276 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.846245 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57a0768-9143-4c87-8178-41bc14afdd5d-openshift-service-ca\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.846475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.846287 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-metrics-certs\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.846475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.846309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.846475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.846340 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdhk\" (UniqueName: \"kubernetes.io/projected/b57a0768-9143-4c87-8178-41bc14afdd5d-kube-api-access-vsdhk\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.947084 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.947045 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57a0768-9143-4c87-8178-41bc14afdd5d-openshift-service-ca\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.947260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.947094 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-metrics-certs\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.947260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.947116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.947260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.947152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdhk\" (UniqueName: \"kubernetes.io/projected/b57a0768-9143-4c87-8178-41bc14afdd5d-kube-api-access-vsdhk\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.947371 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:47:01.947275 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-serving-cert: secret "maas-api-e2e-derive-9e4e10ed-serving-cert" not found Sep 17 17:47:01.947371 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:47:01.947354 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls podName:b57a0768-9143-4c87-8178-41bc14afdd5d nodeName:}" failed. No retries permitted until 2026-09-17 17:47:02.447333291 +0000 UTC m=+959.009498141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls") pod "maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" (UID: "b57a0768-9143-4c87-8178-41bc14afdd5d") : secret "maas-api-e2e-derive-9e4e10ed-serving-cert" not found Sep 17 17:47:01.947710 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.947689 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57a0768-9143-4c87-8178-41bc14afdd5d-openshift-service-ca\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.949549 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.949529 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-metrics-certs\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:01.957492 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:01.957464 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdhk\" (UniqueName: \"kubernetes.io/projected/b57a0768-9143-4c87-8178-41bc14afdd5d-kube-api-access-vsdhk\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:02.451192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.451146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:02.454091 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.454066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls\") pod \"maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:02.658842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.658810 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:02.734376 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.734342 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk"] Sep 17 17:47:02.739143 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.738803 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.742220 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.742124 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9e4e10ed\"" Sep 17 17:47:02.742673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.742656 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9e4e10ed-dockercfg-b2msr\"" Sep 17 17:47:02.753401 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.753320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef258b4a-0604-40d3-b007-bc2a45cd5818-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.753401 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.753353 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hfh\" (UniqueName: \"kubernetes.io/projected/ef258b4a-0604-40d3-b007-bc2a45cd5818-kube-api-access-f7hfh\") pod \"payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.756403 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.756352 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk"] Sep 17 17:47:02.829969 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.829925 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp"] Sep 17 17:47:02.833470 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.833446 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:02.850030 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.850004 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp"] Sep 17 17:47:02.854551 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.854521 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef258b4a-0604-40d3-b007-bc2a45cd5818-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.854740 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.854562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hfh\" (UniqueName: \"kubernetes.io/projected/ef258b4a-0604-40d3-b007-bc2a45cd5818-kube-api-access-f7hfh\") pod \"payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.854740 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.854611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425qm\" (UniqueName: \"kubernetes.io/projected/0f99bf07-5d5e-4341-b464-b675d140a457-kube-api-access-425qm\") pod \"payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:02.854872 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.854774 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f99bf07-5d5e-4341-b464-b675d140a457-plugins-config-volume\") pod \"payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:02.855439 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.855421 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef258b4a-0604-40d3-b007-bc2a45cd5818-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.881632 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.881598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hfh\" (UniqueName: \"kubernetes.io/projected/ef258b4a-0604-40d3-b007-bc2a45cd5818-kube-api-access-f7hfh\") pod \"payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:02.956192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.956149 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f99bf07-5d5e-4341-b464-b675d140a457-plugins-config-volume\") pod \"payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:02.956380 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.956243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425qm\" (UniqueName: \"kubernetes.io/projected/0f99bf07-5d5e-4341-b464-b675d140a457-kube-api-access-425qm\") pod \"payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:02.957038 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.956948 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f99bf07-5d5e-4341-b464-b675d140a457-plugins-config-volume\") pod \"payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:02.967108 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.967084 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm"] Sep 17 17:47:02.968787 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:47:02.968760 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57a0768_9143_4c87_8178_41bc14afdd5d.slice/crio-1659d3ac430b2206fae76c774b64eb9f8b6a0cdb449d2caed040b6c26482d0d1 WatchSource:0}: Error finding container 1659d3ac430b2206fae76c774b64eb9f8b6a0cdb449d2caed040b6c26482d0d1: Status 404 returned error can't find the container with id 1659d3ac430b2206fae76c774b64eb9f8b6a0cdb449d2caed040b6c26482d0d1 Sep 17 17:47:02.977793 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:02.977764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-425qm\" (UniqueName: \"kubernetes.io/projected/0f99bf07-5d5e-4341-b464-b675d140a457-kube-api-access-425qm\") pod \"payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:03.049463 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:03.049427 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:03.144207 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:03.144175 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:03.246374 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:03.246282 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" event={"ID":"b57a0768-9143-4c87-8178-41bc14afdd5d","Type":"ContainerStarted","Data":"1659d3ac430b2206fae76c774b64eb9f8b6a0cdb449d2caed040b6c26482d0d1"} Sep 17 17:47:03.324947 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:03.324848 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk"] Sep 17 17:47:03.326994 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:47:03.326949 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef258b4a_0604_40d3_b007_bc2a45cd5818.slice/crio-35b201055e440a692a1039f328d478ed47c316ed829bbf81250ebb29e61d59bb WatchSource:0}: Error finding container 35b201055e440a692a1039f328d478ed47c316ed829bbf81250ebb29e61d59bb: Status 404 returned error can't find the container with id 35b201055e440a692a1039f328d478ed47c316ed829bbf81250ebb29e61d59bb Sep 17 17:47:03.421055 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:03.421032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp"] Sep 17 17:47:04.252608 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.252517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" event={"ID":"b57a0768-9143-4c87-8178-41bc14afdd5d","Type":"ContainerStarted","Data":"9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186"} Sep 17 17:47:04.253034 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.252696 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:04.254255 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.254225 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" event={"ID":"0f99bf07-5d5e-4341-b464-b675d140a457","Type":"ContainerStarted","Data":"7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb"} Sep 17 17:47:04.254370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.254277 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:04.254370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.254299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" event={"ID":"0f99bf07-5d5e-4341-b464-b675d140a457","Type":"ContainerStarted","Data":"c0685ebc7e84e59403ba5e3dc83c212aa4c81040c71c4e91ec092f5dccd48503"} Sep 17 17:47:04.255607 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.255586 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" event={"ID":"ef258b4a-0604-40d3-b007-bc2a45cd5818","Type":"ContainerStarted","Data":"122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d"} Sep 17 17:47:04.255695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.255624 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:04.255695 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.255640 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" event={"ID":"ef258b4a-0604-40d3-b007-bc2a45cd5818","Type":"ContainerStarted","Data":"35b201055e440a692a1039f328d478ed47c316ed829bbf81250ebb29e61d59bb"} Sep 17 17:47:04.274205 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.274158 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" podStartSLOduration=2.869118478 podStartE2EDuration="3.274144465s" totalImagesPullingTime="405.025987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:04.273238159 +0000 UTC m=+960.835403021" watchObservedRunningTime="2026-09-17 17:47:04.274144465 +0000 UTC m=+960.836309324" Sep 17 17:47:04.298396 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.298342 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" podStartSLOduration=2.298322225 podStartE2EDuration="2.298322225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:04.297541894 +0000 UTC m=+960.859706766" watchObservedRunningTime="2026-09-17 17:47:04.298322225 +0000 UTC m=+960.860487097" Sep 17 17:47:04.323770 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:04.323719 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" podStartSLOduration=2.323704005 podStartE2EDuration="2.323704005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:04.319315972 +0000 UTC m=+960.881480832" watchObservedRunningTime="2026-09-17 17:47:04.323704005 +0000 UTC m=+960.885868864" Sep 17 17:47:10.265940 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:10.265904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:14.738339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.738304 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf"] Sep 17 17:47:14.743038 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.743021 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.746154 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.746132 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-af14f8\"/\"kube-root-ca.crt\"" Sep 17 17:47:14.747032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.747005 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-af14f8\"/\"autores-model-af14f8-kserve-self-signed-certs\"" Sep 17 17:47:14.747032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.747026 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-af14f8\"/\"openshift-service-ca.crt\"" Sep 17 17:47:14.747211 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.747145 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-af14f8\"/\"default-dockercfg-22tnp\"" Sep 17 17:47:14.754432 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.754412 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf"] Sep 17 17:47:14.861026 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.860994 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-home\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.861026 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.861028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7pq\" (UniqueName: \"kubernetes.io/projected/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-kube-api-access-8c7pq\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.861253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.861064 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tls-certs\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.861253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.861130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-model-cache\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.861253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.861156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-dshm\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.861253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.861179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tmp-dir\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.962673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.962647 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-home\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.962673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.962675 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7pq\" (UniqueName: \"kubernetes.io/projected/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-kube-api-access-8c7pq\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.962947 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.962709 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tls-certs\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.962947 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.962730 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-model-cache\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.962947 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.962817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-dshm\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.962947 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.962863 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tmp-dir\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.963185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.963125 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-home\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.963185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.963149 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-model-cache\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.963332 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.963302 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tmp-dir\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.965005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.964978 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-dshm\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.965321 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.965302 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tls-certs\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:14.972940 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:14.972915 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7pq\" (UniqueName: \"kubernetes.io/projected/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-kube-api-access-8c7pq\") pod \"autores-model-af14f8-kserve-df7cfb5f5-9n8gf\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:15.051662 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:15.051637 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:15.263464 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:15.263437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:15.263645 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:15.263523 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:15.326248 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:15.326219 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf"] Sep 17 17:47:15.327551 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:47:15.327524 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec9dcf28_2218_4f51_89c1_e42ef5a4162b.slice/crio-a5afae0616406f47e73182f646b5e3bfdda969efeabe1ec266bf02945e9a9ca6 WatchSource:0}: Error finding container a5afae0616406f47e73182f646b5e3bfdda969efeabe1ec266bf02945e9a9ca6: Status 404 returned error can't find the container with id a5afae0616406f47e73182f646b5e3bfdda969efeabe1ec266bf02945e9a9ca6 Sep 17 17:47:16.322564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:16.322519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" event={"ID":"ec9dcf28-2218-4f51-89c1-e42ef5a4162b","Type":"ContainerStarted","Data":"6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755"} Sep 17 17:47:16.322564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:16.322570 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" event={"ID":"ec9dcf28-2218-4f51-89c1-e42ef5a4162b","Type":"ContainerStarted","Data":"a5afae0616406f47e73182f646b5e3bfdda969efeabe1ec266bf02945e9a9ca6"} Sep 17 17:47:16.343603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:16.343556 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" podStartSLOduration=2.343541628 podStartE2EDuration="2.343541628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:16.341824546 +0000 UTC m=+972.903989400" watchObservedRunningTime="2026-09-17 17:47:16.343541628 +0000 UTC m=+972.905706488" Sep 17 17:47:25.052765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:25.052728 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:25.053331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:25.053040 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:25.065829 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:25.065806 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:25.385577 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:25.385488 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:47:41.445478 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.445444 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm"] Sep 17 17:47:41.446619 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.446419 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" podUID="b57a0768-9143-4c87-8178-41bc14afdd5d" containerName="maas-api" containerID="cri-o://9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186" gracePeriod=30 Sep 17 17:47:41.528333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.528302 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp"] Sep 17 17:47:41.528528 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.528492 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" podUID="0f99bf07-5d5e-4341-b464-b675d140a457" containerName="payload-processing" containerID="cri-o://7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb" gracePeriod=30 Sep 17 17:47:41.539371 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.539345 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk"] Sep 17 17:47:41.539548 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.539528 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" podUID="ef258b4a-0604-40d3-b007-bc2a45cd5818" containerName="payload-pre-processing" containerID="cri-o://122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d" gracePeriod=30 Sep 17 17:47:41.805308 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.805283 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:41.852621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.852587 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:41.876769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.876743 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:41.930723 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.930686 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f99bf07-5d5e-4341-b464-b675d140a457-plugins-config-volume\") pod \"0f99bf07-5d5e-4341-b464-b675d140a457\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " Sep 17 17:47:41.930893 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.930780 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-425qm\" (UniqueName: \"kubernetes.io/projected/0f99bf07-5d5e-4341-b464-b675d140a457-kube-api-access-425qm\") pod \"0f99bf07-5d5e-4341-b464-b675d140a457\" (UID: \"0f99bf07-5d5e-4341-b464-b675d140a457\") " Sep 17 17:47:41.931139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.931115 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f99bf07-5d5e-4341-b464-b675d140a457-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f99bf07-5d5e-4341-b464-b675d140a457" (UID: "0f99bf07-5d5e-4341-b464-b675d140a457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:47:41.932804 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:41.932780 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f99bf07-5d5e-4341-b464-b675d140a457-kube-api-access-425qm" (OuterVolumeSpecName: "kube-api-access-425qm") pod "0f99bf07-5d5e-4341-b464-b675d140a457" (UID: "0f99bf07-5d5e-4341-b464-b675d140a457"). InnerVolumeSpecName "kube-api-access-425qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:47:42.031975 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.031872 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls\") pod \"b57a0768-9143-4c87-8178-41bc14afdd5d\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " Sep 17 17:47:42.031975 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.031913 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsdhk\" (UniqueName: \"kubernetes.io/projected/b57a0768-9143-4c87-8178-41bc14afdd5d-kube-api-access-vsdhk\") pod \"b57a0768-9143-4c87-8178-41bc14afdd5d\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " Sep 17 17:47:42.031975 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.031939 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7hfh\" (UniqueName: \"kubernetes.io/projected/ef258b4a-0604-40d3-b007-bc2a45cd5818-kube-api-access-f7hfh\") pod \"ef258b4a-0604-40d3-b007-bc2a45cd5818\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " Sep 17 17:47:42.032202 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.031985 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-metrics-certs\") pod \"b57a0768-9143-4c87-8178-41bc14afdd5d\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " Sep 17 17:47:42.032202 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.032057 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef258b4a-0604-40d3-b007-bc2a45cd5818-plugins-config-volume\") pod \"ef258b4a-0604-40d3-b007-bc2a45cd5818\" (UID: \"ef258b4a-0604-40d3-b007-bc2a45cd5818\") " Sep 17 17:47:42.032202 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.032134 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57a0768-9143-4c87-8178-41bc14afdd5d-openshift-service-ca\") pod \"b57a0768-9143-4c87-8178-41bc14afdd5d\" (UID: \"b57a0768-9143-4c87-8178-41bc14afdd5d\") " Sep 17 17:47:42.032456 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.032433 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425qm\" (UniqueName: \"kubernetes.io/projected/0f99bf07-5d5e-4341-b464-b675d140a457-kube-api-access-425qm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.032566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.032459 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f99bf07-5d5e-4341-b464-b675d140a457-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.032566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.032500 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef258b4a-0604-40d3-b007-bc2a45cd5818-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef258b4a-0604-40d3-b007-bc2a45cd5818" (UID: "ef258b4a-0604-40d3-b007-bc2a45cd5818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:47:42.032566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.032536 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57a0768-9143-4c87-8178-41bc14afdd5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b57a0768-9143-4c87-8178-41bc14afdd5d" (UID: "b57a0768-9143-4c87-8178-41bc14afdd5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:47:42.034161 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.034131 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b57a0768-9143-4c87-8178-41bc14afdd5d" (UID: "b57a0768-9143-4c87-8178-41bc14afdd5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:47:42.034356 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.034332 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57a0768-9143-4c87-8178-41bc14afdd5d-kube-api-access-vsdhk" (OuterVolumeSpecName: "kube-api-access-vsdhk") pod "b57a0768-9143-4c87-8178-41bc14afdd5d" (UID: "b57a0768-9143-4c87-8178-41bc14afdd5d"). InnerVolumeSpecName "kube-api-access-vsdhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:47:42.034450 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.034428 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef258b4a-0604-40d3-b007-bc2a45cd5818-kube-api-access-f7hfh" (OuterVolumeSpecName: "kube-api-access-f7hfh") pod "ef258b4a-0604-40d3-b007-bc2a45cd5818" (UID: "ef258b4a-0604-40d3-b007-bc2a45cd5818"). InnerVolumeSpecName "kube-api-access-f7hfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:47:42.034506 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.034460 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b57a0768-9143-4c87-8178-41bc14afdd5d" (UID: "b57a0768-9143-4c87-8178-41bc14afdd5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:47:42.133059 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.133026 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hfh\" (UniqueName: \"kubernetes.io/projected/ef258b4a-0604-40d3-b007-bc2a45cd5818-kube-api-access-f7hfh\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.133059 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.133052 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.133059 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.133062 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef258b4a-0604-40d3-b007-bc2a45cd5818-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.133278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.133072 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57a0768-9143-4c87-8178-41bc14afdd5d-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.133278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.133081 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57a0768-9143-4c87-8178-41bc14afdd5d-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.133278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.133088 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsdhk\" (UniqueName: \"kubernetes.io/projected/b57a0768-9143-4c87-8178-41bc14afdd5d-kube-api-access-vsdhk\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:47:42.470912 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.470877 2566 generic.go:356] "Generic (PLEG): container finished" podID="b57a0768-9143-4c87-8178-41bc14afdd5d" containerID="9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186" exitCode=0 Sep 17 17:47:42.471349 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.470924 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" Sep 17 17:47:42.471349 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.470950 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" event={"ID":"b57a0768-9143-4c87-8178-41bc14afdd5d","Type":"ContainerDied","Data":"9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186"} Sep 17 17:47:42.471349 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.471035 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm" event={"ID":"b57a0768-9143-4c87-8178-41bc14afdd5d","Type":"ContainerDied","Data":"1659d3ac430b2206fae76c774b64eb9f8b6a0cdb449d2caed040b6c26482d0d1"} Sep 17 17:47:42.471349 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.471079 2566 scope.go:117] "RemoveContainer" containerID="9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186" Sep 17 17:47:42.472554 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.472531 2566 generic.go:356] "Generic (PLEG): container finished" podID="0f99bf07-5d5e-4341-b464-b675d140a457" containerID="7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb" exitCode=0 Sep 17 17:47:42.472621 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.472589 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" Sep 17 17:47:42.472727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.472672 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" event={"ID":"0f99bf07-5d5e-4341-b464-b675d140a457","Type":"ContainerDied","Data":"7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb"} Sep 17 17:47:42.472727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.472720 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp" event={"ID":"0f99bf07-5d5e-4341-b464-b675d140a457","Type":"ContainerDied","Data":"c0685ebc7e84e59403ba5e3dc83c212aa4c81040c71c4e91ec092f5dccd48503"} Sep 17 17:47:42.481569 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.481520 2566 generic.go:356] "Generic (PLEG): container finished" podID="ef258b4a-0604-40d3-b007-bc2a45cd5818" containerID="122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d" exitCode=0 Sep 17 17:47:42.481771 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.481603 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" event={"ID":"ef258b4a-0604-40d3-b007-bc2a45cd5818","Type":"ContainerDied","Data":"122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d"} Sep 17 17:47:42.481771 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.481685 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" event={"ID":"ef258b4a-0604-40d3-b007-bc2a45cd5818","Type":"ContainerDied","Data":"35b201055e440a692a1039f328d478ed47c316ed829bbf81250ebb29e61d59bb"} Sep 17 17:47:42.481991 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.481795 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk" Sep 17 17:47:42.489455 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.489437 2566 scope.go:117] "RemoveContainer" containerID="9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186" Sep 17 17:47:42.489752 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:47:42.489730 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186\": container with ID starting with 9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186 not found: ID does not exist" containerID="9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186" Sep 17 17:47:42.489793 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.489760 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186"} err="failed to get container status \"9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186\": rpc error: code = NotFound desc = could not find container \"9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186\": container with ID starting with 9f0d26b8ec85d65d371731c2223c874e080cbe47c1da9f1b2a64697129f3a186 not found: ID does not exist" Sep 17 17:47:42.489793 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.489777 2566 scope.go:117] "RemoveContainer" containerID="7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb" Sep 17 17:47:42.498595 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.498574 2566 scope.go:117] "RemoveContainer" containerID="7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb" Sep 17 17:47:42.498912 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:47:42.498882 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb\": container with ID starting with 7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb not found: ID does not exist" containerID="7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb" Sep 17 17:47:42.499052 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.498924 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb"} err="failed to get container status \"7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb\": rpc error: code = NotFound desc = could not find container \"7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb\": container with ID starting with 7635581c1f9c2ed7e232d401104d7531819634fe12bfb34121fc0de4da03b3eb not found: ID does not exist" Sep 17 17:47:42.499126 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.498948 2566 scope.go:117] "RemoveContainer" containerID="122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d" Sep 17 17:47:42.506357 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.506324 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm"] Sep 17 17:47:42.509339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.509311 2566 scope.go:117] "RemoveContainer" containerID="122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d" Sep 17 17:47:42.509721 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:47:42.509701 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d\": container with ID starting with 122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d not found: ID does not exist" containerID="122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d" Sep 17 17:47:42.509781 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.509726 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d"} err="failed to get container status \"122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d\": rpc error: code = NotFound desc = could not find container \"122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d\": container with ID starting with 122bd9d7187bc71baa2c8a82281cdf8442b70378d59f91c6e63f73cf1e001d5d not found: ID does not exist" Sep 17 17:47:42.523041 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.523011 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e4e10ed-77d99f976b-dmrmm"] Sep 17 17:47:42.545113 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.545083 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp"] Sep 17 17:47:42.553000 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.552975 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e4e10ed-58db75dbf7-hg8sp"] Sep 17 17:47:42.574590 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.574557 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk"] Sep 17 17:47:42.587816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:42.587784 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e4e10ed-79bd859cc4-98cjk"] Sep 17 17:47:44.150443 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:44.150408 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f99bf07-5d5e-4341-b464-b675d140a457" path="/var/lib/kubelet/pods/0f99bf07-5d5e-4341-b464-b675d140a457/volumes" Sep 17 17:47:44.150879 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:44.150865 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57a0768-9143-4c87-8178-41bc14afdd5d" path="/var/lib/kubelet/pods/b57a0768-9143-4c87-8178-41bc14afdd5d/volumes" Sep 17 17:47:44.151311 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:47:44.151290 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef258b4a-0604-40d3-b007-bc2a45cd5818" path="/var/lib/kubelet/pods/ef258b4a-0604-40d3-b007-bc2a45cd5818/volumes" Sep 17 17:48:00.198612 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:00.198577 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf"] Sep 17 17:48:01.939151 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:01.939114 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6"] Sep 17 17:48:01.939982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:01.939911 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" podUID="29d176e8-6196-4867-a79b-b78b51101671" containerName="payload-pre-processing" containerID="cri-o://ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9" gracePeriod=30 Sep 17 17:48:02.196983 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.196905 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:48:02.304861 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.304829 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-954vk\" (UniqueName: \"kubernetes.io/projected/29d176e8-6196-4867-a79b-b78b51101671-kube-api-access-954vk\") pod \"29d176e8-6196-4867-a79b-b78b51101671\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " Sep 17 17:48:02.305056 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.304872 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d176e8-6196-4867-a79b-b78b51101671-plugins-config-volume\") pod \"29d176e8-6196-4867-a79b-b78b51101671\" (UID: \"29d176e8-6196-4867-a79b-b78b51101671\") " Sep 17 17:48:02.305266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.305243 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d176e8-6196-4867-a79b-b78b51101671-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29d176e8-6196-4867-a79b-b78b51101671" (UID: "29d176e8-6196-4867-a79b-b78b51101671"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:48:02.307054 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.307029 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d176e8-6196-4867-a79b-b78b51101671-kube-api-access-954vk" (OuterVolumeSpecName: "kube-api-access-954vk") pod "29d176e8-6196-4867-a79b-b78b51101671" (UID: "29d176e8-6196-4867-a79b-b78b51101671"). InnerVolumeSpecName "kube-api-access-954vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:48:02.405755 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.405709 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954vk\" (UniqueName: \"kubernetes.io/projected/29d176e8-6196-4867-a79b-b78b51101671-kube-api-access-954vk\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:02.405755 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.405751 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d176e8-6196-4867-a79b-b78b51101671-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:02.594005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.593966 2566 generic.go:356] "Generic (PLEG): container finished" podID="29d176e8-6196-4867-a79b-b78b51101671" containerID="ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9" exitCode=0 Sep 17 17:48:02.594005 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.594005 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" Sep 17 17:48:02.594209 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.594058 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" event={"ID":"29d176e8-6196-4867-a79b-b78b51101671","Type":"ContainerDied","Data":"ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9"} Sep 17 17:48:02.594209 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.594113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6" event={"ID":"29d176e8-6196-4867-a79b-b78b51101671","Type":"ContainerDied","Data":"8b4f5a32c4c10ba13622f307e319d14a767fa837d78a0127c1f1de9d0edefc3a"} Sep 17 17:48:02.594209 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.594138 2566 scope.go:117] "RemoveContainer" containerID="ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9" Sep 17 17:48:02.603155 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.603133 2566 scope.go:117] "RemoveContainer" containerID="ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9" Sep 17 17:48:02.603398 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:48:02.603379 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9\": container with ID starting with ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9 not found: ID does not exist" containerID="ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9" Sep 17 17:48:02.603471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.603410 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9"} err="failed to get container status \"ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9\": rpc error: code = NotFound desc = could not find container \"ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9\": container with ID starting with ca0040903aad2902efa1ba21bfc1f173f199db5b722434cedd0c722077543da9 not found: ID does not exist" Sep 17 17:48:02.631528 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.631501 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6"] Sep 17 17:48:02.641879 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:02.641855 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-af14f8-6f65467b94-dtxg6"] Sep 17 17:48:04.150253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:04.150215 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29d176e8-6196-4867-a79b-b78b51101671" path="/var/lib/kubelet/pods/29d176e8-6196-4867-a79b-b78b51101671/volumes" Sep 17 17:48:14.543090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543059 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6"] Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543416 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d176e8-6196-4867-a79b-b78b51101671" containerName="payload-pre-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543427 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d176e8-6196-4867-a79b-b78b51101671" containerName="payload-pre-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543436 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f99bf07-5d5e-4341-b464-b675d140a457" containerName="payload-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543441 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f99bf07-5d5e-4341-b464-b675d140a457" containerName="payload-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543458 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef258b4a-0604-40d3-b007-bc2a45cd5818" containerName="payload-pre-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543471 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef258b4a-0604-40d3-b007-bc2a45cd5818" containerName="payload-pre-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543481 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57a0768-9143-4c87-8178-41bc14afdd5d" containerName="maas-api" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543486 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57a0768-9143-4c87-8178-41bc14afdd5d" containerName="maas-api" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543551 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef258b4a-0604-40d3-b007-bc2a45cd5818" containerName="payload-pre-processing" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543564 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57a0768-9143-4c87-8178-41bc14afdd5d" containerName="maas-api" Sep 17 17:48:14.543560 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543569 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29d176e8-6196-4867-a79b-b78b51101671" containerName="payload-pre-processing" Sep 17 17:48:14.543936 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.543575 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f99bf07-5d5e-4341-b464-b675d140a457" containerName="payload-processing" Sep 17 17:48:14.548087 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.548067 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.552116 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.552093 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-db7813\"/\"kube-root-ca.crt\"" Sep 17 17:48:14.552234 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.552110 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db7813\"/\"default-dockercfg-rhrfd\"" Sep 17 17:48:14.552234 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.552200 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-db7813\"/\"openshift-service-ca.crt\"" Sep 17 17:48:14.552234 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.552200 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db7813\"/\"premium-w0-db7813-kserve-self-signed-certs\"" Sep 17 17:48:14.561540 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.561516 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6"] Sep 17 17:48:14.618543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.618513 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-dshm\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.618543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.618544 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-home\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.618776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.618571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz56m\" (UniqueName: \"kubernetes.io/projected/38e13467-a5e2-4e01-ad08-b0b837795dcf-kube-api-access-tz56m\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.618776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.618652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-model-cache\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.618776 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.618749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-tmp-dir\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.618917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.618786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e13467-a5e2-4e01-ad08-b0b837795dcf-tls-certs\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.719910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.719872 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-dshm\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.719910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.719913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-home\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720188 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.719941 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz56m\" (UniqueName: \"kubernetes.io/projected/38e13467-a5e2-4e01-ad08-b0b837795dcf-kube-api-access-tz56m\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720188 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.720000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-model-cache\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720188 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.720078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-tmp-dir\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720188 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.720102 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e13467-a5e2-4e01-ad08-b0b837795dcf-tls-certs\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720400 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.720381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-home\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720459 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.720405 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-model-cache\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.720459 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.720446 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-tmp-dir\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.722123 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.722104 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-dshm\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.722473 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.722456 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e13467-a5e2-4e01-ad08-b0b837795dcf-tls-certs\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.729639 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.729614 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz56m\" (UniqueName: \"kubernetes.io/projected/38e13467-a5e2-4e01-ad08-b0b837795dcf-kube-api-access-tz56m\") pod \"premium-w0-db7813-kserve-5498ff8899-pg2k6\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:14.857180 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:14.857145 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:15.148090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.148063 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6"] Sep 17 17:48:15.148527 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:48:15.148500 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38e13467_a5e2_4e01_ad08_b0b837795dcf.slice/crio-8883f2aa37ca12fb58c4b887c92548cea48b08660d76c73d635b0f3bde901c84 WatchSource:0}: Error finding container 8883f2aa37ca12fb58c4b887c92548cea48b08660d76c73d635b0f3bde901c84: Status 404 returned error can't find the container with id 8883f2aa37ca12fb58c4b887c92548cea48b08660d76c73d635b0f3bde901c84 Sep 17 17:48:15.217127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.217091 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" podUID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" containerName="main" containerID="cri-o://6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755" gracePeriod=45 Sep 17 17:48:15.373972 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.373921 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" podUID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Sep 17 17:48:15.452186 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.452163 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:48:15.529244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529215 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tmp-dir\") pod \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " Sep 17 17:48:15.529394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529256 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-home\") pod \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " Sep 17 17:48:15.529394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529308 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7pq\" (UniqueName: \"kubernetes.io/projected/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-kube-api-access-8c7pq\") pod \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " Sep 17 17:48:15.529394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529325 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-model-cache\") pod \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " Sep 17 17:48:15.529394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529350 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tls-certs\") pod \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " Sep 17 17:48:15.529394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529375 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-dshm\") pod \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\" (UID: \"ec9dcf28-2218-4f51-89c1-e42ef5a4162b\") " Sep 17 17:48:15.529628 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529483 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec9dcf28-2218-4f51-89c1-e42ef5a4162b" (UID: "ec9dcf28-2218-4f51-89c1-e42ef5a4162b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:48:15.529628 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529559 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-home" (OuterVolumeSpecName: "home") pod "ec9dcf28-2218-4f51-89c1-e42ef5a4162b" (UID: "ec9dcf28-2218-4f51-89c1-e42ef5a4162b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:48:15.529718 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529660 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:15.529718 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529678 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:15.529818 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.529783 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec9dcf28-2218-4f51-89c1-e42ef5a4162b" (UID: "ec9dcf28-2218-4f51-89c1-e42ef5a4162b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:48:15.531476 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.531448 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-dshm" (OuterVolumeSpecName: "dshm") pod "ec9dcf28-2218-4f51-89c1-e42ef5a4162b" (UID: "ec9dcf28-2218-4f51-89c1-e42ef5a4162b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:48:15.531584 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.531568 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec9dcf28-2218-4f51-89c1-e42ef5a4162b" (UID: "ec9dcf28-2218-4f51-89c1-e42ef5a4162b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:48:15.531626 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.531603 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-kube-api-access-8c7pq" (OuterVolumeSpecName: "kube-api-access-8c7pq") pod "ec9dcf28-2218-4f51-89c1-e42ef5a4162b" (UID: "ec9dcf28-2218-4f51-89c1-e42ef5a4162b"). InnerVolumeSpecName "kube-api-access-8c7pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:48:15.630481 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.630450 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:15.630481 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.630474 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:15.630481 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.630485 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7pq\" (UniqueName: \"kubernetes.io/projected/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-kube-api-access-8c7pq\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:15.630889 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.630493 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9dcf28-2218-4f51-89c1-e42ef5a4162b-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:48:15.659607 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.659571 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" event={"ID":"38e13467-a5e2-4e01-ad08-b0b837795dcf","Type":"ContainerStarted","Data":"8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892"} Sep 17 17:48:15.659769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.659616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" event={"ID":"38e13467-a5e2-4e01-ad08-b0b837795dcf","Type":"ContainerStarted","Data":"8883f2aa37ca12fb58c4b887c92548cea48b08660d76c73d635b0f3bde901c84"} Sep 17 17:48:15.660936 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.660913 2566 generic.go:356] "Generic (PLEG): container finished" podID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" containerID="6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755" exitCode=0 Sep 17 17:48:15.661064 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.660944 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" Sep 17 17:48:15.661064 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.661003 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" event={"ID":"ec9dcf28-2218-4f51-89c1-e42ef5a4162b","Type":"ContainerDied","Data":"6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755"} Sep 17 17:48:15.661064 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.661049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf" event={"ID":"ec9dcf28-2218-4f51-89c1-e42ef5a4162b","Type":"ContainerDied","Data":"a5afae0616406f47e73182f646b5e3bfdda969efeabe1ec266bf02945e9a9ca6"} Sep 17 17:48:15.661224 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.661074 2566 scope.go:117] "RemoveContainer" containerID="6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755" Sep 17 17:48:15.670650 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.670627 2566 scope.go:117] "RemoveContainer" containerID="6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755" Sep 17 17:48:15.670919 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:48:15.670900 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755\": container with ID starting with 6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755 not found: ID does not exist" containerID="6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755" Sep 17 17:48:15.671020 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.670927 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755"} err="failed to get container status \"6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755\": rpc error: code = NotFound desc = could not find container \"6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755\": container with ID starting with 6312343094a46db743adb564ad3509aae1394f7a4313b2be84b882be12de9755 not found: ID does not exist" Sep 17 17:48:15.681491 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.681449 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" podStartSLOduration=1.681436047 podStartE2EDuration="1.681436047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:15.678704668 +0000 UTC m=+1032.240869528" watchObservedRunningTime="2026-09-17 17:48:15.681436047 +0000 UTC m=+1032.243600907" Sep 17 17:48:15.701397 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.701339 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf"] Sep 17 17:48:15.706293 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:15.706272 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-af14f8/autores-model-af14f8-kserve-df7cfb5f5-9n8gf"] Sep 17 17:48:16.150014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:16.149979 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" path="/var/lib/kubelet/pods/ec9dcf28-2218-4f51-89c1-e42ef5a4162b/volumes" Sep 17 17:48:24.857719 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:24.857687 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:24.858142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:24.857732 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:24.870076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:24.870051 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:25.728181 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:25.728151 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:48:53.592023 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.591989 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf"] Sep 17 17:48:53.592471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.592367 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" containerName="main" Sep 17 17:48:53.592471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.592377 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" containerName="main" Sep 17 17:48:53.592471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.592446 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9dcf28-2218-4f51-89c1-e42ef5a4162b" containerName="main" Sep 17 17:48:53.595717 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.595701 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.598572 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.598552 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6239ea-dockercfg-fql2m\"" Sep 17 17:48:53.599033 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.599017 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6239ea\"" Sep 17 17:48:53.612187 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.612163 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf"] Sep 17 17:48:53.653086 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.653055 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh"] Sep 17 17:48:53.656573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.656552 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.671414 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.671388 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh"] Sep 17 17:48:53.671554 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.671472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8ed546-b55a-4c9a-b239-1b230ca267fa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.671631 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.671586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxgg\" (UniqueName: \"kubernetes.io/projected/3f8ed546-b55a-4c9a-b239-1b230ca267fa-kube-api-access-9pxgg\") pod \"payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.772768 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.772734 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxgg\" (UniqueName: \"kubernetes.io/projected/3f8ed546-b55a-4c9a-b239-1b230ca267fa-kube-api-access-9pxgg\") pod \"payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.772941 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.772792 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts94l\" (UniqueName: \"kubernetes.io/projected/f1da5568-6019-40c2-a850-452b1f0c2952-kube-api-access-ts94l\") pod \"payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.772941 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.772824 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8ed546-b55a-4c9a-b239-1b230ca267fa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.772941 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.772853 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1da5568-6019-40c2-a850-452b1f0c2952-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.773757 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.773717 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8ed546-b55a-4c9a-b239-1b230ca267fa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.782546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.782521 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxgg\" (UniqueName: \"kubernetes.io/projected/3f8ed546-b55a-4c9a-b239-1b230ca267fa-kube-api-access-9pxgg\") pod \"payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.873763 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.873667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts94l\" (UniqueName: \"kubernetes.io/projected/f1da5568-6019-40c2-a850-452b1f0c2952-kube-api-access-ts94l\") pod \"payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.873763 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.873743 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1da5568-6019-40c2-a850-452b1f0c2952-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.874469 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.874444 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1da5568-6019-40c2-a850-452b1f0c2952-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.884244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.884213 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts94l\" (UniqueName: \"kubernetes.io/projected/f1da5568-6019-40c2-a850-452b1f0c2952-kube-api-access-ts94l\") pod \"payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:53.905897 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.905870 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:53.967331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:53.966605 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:54.179410 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.179315 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf"] Sep 17 17:48:54.184511 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:48:54.184481 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8ed546_b55a_4c9a_b239_1b230ca267fa.slice/crio-7ed40e045e52c63b822fc862b48be3c813042a6acf8f95ecbfb00e81e3cfec23 WatchSource:0}: Error finding container 7ed40e045e52c63b822fc862b48be3c813042a6acf8f95ecbfb00e81e3cfec23: Status 404 returned error can't find the container with id 7ed40e045e52c63b822fc862b48be3c813042a6acf8f95ecbfb00e81e3cfec23 Sep 17 17:48:54.258553 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.258459 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh"] Sep 17 17:48:54.261087 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:48:54.261056 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1da5568_6019_40c2_a850_452b1f0c2952.slice/crio-f9fff6b2117a525bca55e36c69d571051e80517827913b1987079644ef7bc973 WatchSource:0}: Error finding container f9fff6b2117a525bca55e36c69d571051e80517827913b1987079644ef7bc973: Status 404 returned error can't find the container with id f9fff6b2117a525bca55e36c69d571051e80517827913b1987079644ef7bc973 Sep 17 17:48:54.867389 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.867353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" event={"ID":"f1da5568-6019-40c2-a850-452b1f0c2952","Type":"ContainerStarted","Data":"23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a"} Sep 17 17:48:54.867848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.867414 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" event={"ID":"f1da5568-6019-40c2-a850-452b1f0c2952","Type":"ContainerStarted","Data":"f9fff6b2117a525bca55e36c69d571051e80517827913b1987079644ef7bc973"} Sep 17 17:48:54.867848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.867444 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:48:54.869565 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.869531 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" event={"ID":"3f8ed546-b55a-4c9a-b239-1b230ca267fa","Type":"ContainerStarted","Data":"ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08"} Sep 17 17:48:54.869704 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.869580 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" event={"ID":"3f8ed546-b55a-4c9a-b239-1b230ca267fa","Type":"ContainerStarted","Data":"7ed40e045e52c63b822fc862b48be3c813042a6acf8f95ecbfb00e81e3cfec23"} Sep 17 17:48:54.870067 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.870048 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:48:54.888982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.888234 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" podStartSLOduration=1.8882158530000002 podStartE2EDuration="1.888215853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:54.884309531 +0000 UTC m=+1071.446474392" watchObservedRunningTime="2026-09-17 17:48:54.888215853 +0000 UTC m=+1071.450380717" Sep 17 17:48:54.904755 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:48:54.904710 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" podStartSLOduration=1.904696172 podStartE2EDuration="1.904696172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:54.903179281 +0000 UTC m=+1071.465344145" watchObservedRunningTime="2026-09-17 17:48:54.904696172 +0000 UTC m=+1071.466861034" Sep 17 17:49:05.826190 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.826156 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h"] Sep 17 17:49:05.830761 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.830743 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:05.833380 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.833357 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"premium-w4-8aeea1-kserve-self-signed-certs\"" Sep 17 17:49:05.845026 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.845000 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h"] Sep 17 17:49:05.876886 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.876854 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:49:05.985566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.985534 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337d2a80-65f5-42a5-a841-44127c42b351-tls-certs\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:05.985741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.985580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-home\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:05.985741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.985650 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-model-cache\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:05.985741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.985719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mb2z\" (UniqueName: \"kubernetes.io/projected/337d2a80-65f5-42a5-a841-44127c42b351-kube-api-access-6mb2z\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:05.985878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.985773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-tmp-dir\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:05.985878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:05.985814 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-dshm\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.086798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.086713 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337d2a80-65f5-42a5-a841-44127c42b351-tls-certs\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.086798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.086747 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-home\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.086798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.086763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-model-cache\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.086798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.086784 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mb2z\" (UniqueName: \"kubernetes.io/projected/337d2a80-65f5-42a5-a841-44127c42b351-kube-api-access-6mb2z\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.086798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.086803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-tmp-dir\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.087228 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.086824 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-dshm\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.087228 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.087172 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-tmp-dir\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.087326 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.087229 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-model-cache\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.087326 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.087299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-home\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.089079 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.089053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-dshm\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.089489 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.089462 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337d2a80-65f5-42a5-a841-44127c42b351-tls-certs\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.097099 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.097076 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mb2z\" (UniqueName: \"kubernetes.io/projected/337d2a80-65f5-42a5-a841-44127c42b351-kube-api-access-6mb2z\") pod \"premium-w4-8aeea1-kserve-68f78f8789-cms6h\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.140199 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.140173 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:06.421732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.421660 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h"] Sep 17 17:49:06.425098 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:49:06.425068 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod337d2a80_65f5_42a5_a841_44127c42b351.slice/crio-1b1a2dfbb01dce8aff7b7e4e7d09a8ddb71ee4896f6b4a260d67e83939ae8d89 WatchSource:0}: Error finding container 1b1a2dfbb01dce8aff7b7e4e7d09a8ddb71ee4896f6b4a260d67e83939ae8d89: Status 404 returned error can't find the container with id 1b1a2dfbb01dce8aff7b7e4e7d09a8ddb71ee4896f6b4a260d67e83939ae8d89 Sep 17 17:49:06.882985 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.882940 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:49:06.936060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.935983 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" event={"ID":"337d2a80-65f5-42a5-a841-44127c42b351","Type":"ContainerStarted","Data":"92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e"} Sep 17 17:49:06.936060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.936036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" event={"ID":"337d2a80-65f5-42a5-a841-44127c42b351","Type":"ContainerStarted","Data":"1b1a2dfbb01dce8aff7b7e4e7d09a8ddb71ee4896f6b4a260d67e83939ae8d89"} Sep 17 17:49:06.959051 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:06.958996 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" podStartSLOduration=1.95897511 podStartE2EDuration="1.95897511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:06.956116196 +0000 UTC m=+1083.518281069" watchObservedRunningTime="2026-09-17 17:49:06.95897511 +0000 UTC m=+1083.521139972" Sep 17 17:49:16.140570 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:16.140532 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:16.141071 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:16.140680 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:16.153295 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:16.153273 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:17.001657 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:17.001622 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:49:38.783166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.783128 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2"] Sep 17 17:49:38.786531 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.786515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:38.788941 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.788920 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8f6c1ea9\"" Sep 17 17:49:38.800139 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.800115 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2"] Sep 17 17:49:38.879482 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.879446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9gz\" (UniqueName: \"kubernetes.io/projected/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-kube-api-access-7q9gz\") pod \"payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:38.879641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.879520 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:38.980998 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.980948 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9gz\" (UniqueName: \"kubernetes.io/projected/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-kube-api-access-7q9gz\") pod \"payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:38.981201 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.981055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:38.981765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.981744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:38.989602 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:38.989583 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9gz\" (UniqueName: \"kubernetes.io/projected/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-kube-api-access-7q9gz\") pod \"payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:39.096991 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:39.096950 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:39.381497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:39.381358 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2"] Sep 17 17:49:39.384198 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:49:39.384166 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ffaf8c_3048_40b3_aab1_9eeaf0d393dc.slice/crio-2fd0a5544cc1aab5f38e17de908f4e21cbd6b9f83660c6ec3f2f83d010a5c2ac WatchSource:0}: Error finding container 2fd0a5544cc1aab5f38e17de908f4e21cbd6b9f83660c6ec3f2f83d010a5c2ac: Status 404 returned error can't find the container with id 2fd0a5544cc1aab5f38e17de908f4e21cbd6b9f83660c6ec3f2f83d010a5c2ac Sep 17 17:49:40.108026 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:40.107975 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" event={"ID":"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc","Type":"ContainerStarted","Data":"f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f"} Sep 17 17:49:40.108490 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:40.108042 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" event={"ID":"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc","Type":"ContainerStarted","Data":"2fd0a5544cc1aab5f38e17de908f4e21cbd6b9f83660c6ec3f2f83d010a5c2ac"} Sep 17 17:49:40.108490 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:40.108084 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:49:40.128376 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:40.128323 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" podStartSLOduration=2.128304129 podStartE2EDuration="2.128304129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:40.124910946 +0000 UTC m=+1116.687075806" watchObservedRunningTime="2026-09-17 17:49:40.128304129 +0000 UTC m=+1116.690468990" Sep 17 17:49:51.115787 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:49:51.115757 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:50:21.459826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.459788 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf"] Sep 17 17:50:21.460349 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.460124 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" podUID="3f8ed546-b55a-4c9a-b239-1b230ca267fa" containerName="payload-processing" containerID="cri-o://ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08" gracePeriod=30 Sep 17 17:50:21.484347 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.484316 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh"] Sep 17 17:50:21.484598 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.484551 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" podUID="f1da5568-6019-40c2-a850-452b1f0c2952" containerName="payload-pre-processing" containerID="cri-o://23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a" gracePeriod=30 Sep 17 17:50:21.740884 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.740859 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:50:21.757021 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.756997 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pxgg\" (UniqueName: \"kubernetes.io/projected/3f8ed546-b55a-4c9a-b239-1b230ca267fa-kube-api-access-9pxgg\") pod \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " Sep 17 17:50:21.757156 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.757041 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8ed546-b55a-4c9a-b239-1b230ca267fa-plugins-config-volume\") pod \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\" (UID: \"3f8ed546-b55a-4c9a-b239-1b230ca267fa\") " Sep 17 17:50:21.757433 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.757412 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8ed546-b55a-4c9a-b239-1b230ca267fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f8ed546-b55a-4c9a-b239-1b230ca267fa" (UID: "3f8ed546-b55a-4c9a-b239-1b230ca267fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:50:21.759201 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.759177 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8ed546-b55a-4c9a-b239-1b230ca267fa-kube-api-access-9pxgg" (OuterVolumeSpecName: "kube-api-access-9pxgg") pod "3f8ed546-b55a-4c9a-b239-1b230ca267fa" (UID: "3f8ed546-b55a-4c9a-b239-1b230ca267fa"). InnerVolumeSpecName "kube-api-access-9pxgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:50:21.771185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.771166 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:50:21.857754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.857723 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1da5568-6019-40c2-a850-452b1f0c2952-plugins-config-volume\") pod \"f1da5568-6019-40c2-a850-452b1f0c2952\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " Sep 17 17:50:21.857754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.857758 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts94l\" (UniqueName: \"kubernetes.io/projected/f1da5568-6019-40c2-a850-452b1f0c2952-kube-api-access-ts94l\") pod \"f1da5568-6019-40c2-a850-452b1f0c2952\" (UID: \"f1da5568-6019-40c2-a850-452b1f0c2952\") " Sep 17 17:50:21.858095 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.858074 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1da5568-6019-40c2-a850-452b1f0c2952-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1da5568-6019-40c2-a850-452b1f0c2952" (UID: "f1da5568-6019-40c2-a850-452b1f0c2952"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:50:21.858145 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.858085 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pxgg\" (UniqueName: \"kubernetes.io/projected/3f8ed546-b55a-4c9a-b239-1b230ca267fa-kube-api-access-9pxgg\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:50:21.858145 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.858118 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8ed546-b55a-4c9a-b239-1b230ca267fa-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:50:21.859856 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.859837 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1da5568-6019-40c2-a850-452b1f0c2952-kube-api-access-ts94l" (OuterVolumeSpecName: "kube-api-access-ts94l") pod "f1da5568-6019-40c2-a850-452b1f0c2952" (UID: "f1da5568-6019-40c2-a850-452b1f0c2952"). InnerVolumeSpecName "kube-api-access-ts94l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:50:21.958654 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.958624 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1da5568-6019-40c2-a850-452b1f0c2952-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:50:21.958654 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:21.958646 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts94l\" (UniqueName: \"kubernetes.io/projected/f1da5568-6019-40c2-a850-452b1f0c2952-kube-api-access-ts94l\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:50:22.319699 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.319668 2566 generic.go:356] "Generic (PLEG): container finished" podID="f1da5568-6019-40c2-a850-452b1f0c2952" containerID="23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a" exitCode=0 Sep 17 17:50:22.319891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.319716 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" Sep 17 17:50:22.319891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.319738 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" event={"ID":"f1da5568-6019-40c2-a850-452b1f0c2952","Type":"ContainerDied","Data":"23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a"} Sep 17 17:50:22.319891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.319780 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh" event={"ID":"f1da5568-6019-40c2-a850-452b1f0c2952","Type":"ContainerDied","Data":"f9fff6b2117a525bca55e36c69d571051e80517827913b1987079644ef7bc973"} Sep 17 17:50:22.319891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.319801 2566 scope.go:117] "RemoveContainer" containerID="23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a" Sep 17 17:50:22.321244 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.321221 2566 generic.go:356] "Generic (PLEG): container finished" podID="3f8ed546-b55a-4c9a-b239-1b230ca267fa" containerID="ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08" exitCode=0 Sep 17 17:50:22.321368 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.321259 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" Sep 17 17:50:22.321368 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.321292 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" event={"ID":"3f8ed546-b55a-4c9a-b239-1b230ca267fa","Type":"ContainerDied","Data":"ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08"} Sep 17 17:50:22.321368 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.321338 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf" event={"ID":"3f8ed546-b55a-4c9a-b239-1b230ca267fa","Type":"ContainerDied","Data":"7ed40e045e52c63b822fc862b48be3c813042a6acf8f95ecbfb00e81e3cfec23"} Sep 17 17:50:22.328719 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.328704 2566 scope.go:117] "RemoveContainer" containerID="23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a" Sep 17 17:50:22.329082 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:50:22.329061 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a\": container with ID starting with 23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a not found: ID does not exist" containerID="23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a" Sep 17 17:50:22.329174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.329089 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a"} err="failed to get container status \"23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a\": rpc error: code = NotFound desc = could not find container \"23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a\": container with ID starting with 23c9da95c8ef8a4c93e15898efa52812e9fdf6b8eac2f0c1676637221ce68e9a not found: ID does not exist" Sep 17 17:50:22.329174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.329104 2566 scope.go:117] "RemoveContainer" containerID="ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08" Sep 17 17:50:22.336797 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.336777 2566 scope.go:117] "RemoveContainer" containerID="ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08" Sep 17 17:50:22.337060 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:50:22.337040 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08\": container with ID starting with ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08 not found: ID does not exist" containerID="ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08" Sep 17 17:50:22.337118 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.337067 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08"} err="failed to get container status \"ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08\": rpc error: code = NotFound desc = could not find container \"ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08\": container with ID starting with ab484f6bcc3f9b6edb01f06e442bdef2ab067f6e6090aca16c9d9131c52f3d08 not found: ID does not exist" Sep 17 17:50:22.340521 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.340501 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf"] Sep 17 17:50:22.344805 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.344785 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6239ea-56cb76d78f-znnkf"] Sep 17 17:50:22.360597 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.360574 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh"] Sep 17 17:50:22.367713 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:22.367695 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6239ea-776dd88474-vfchh"] Sep 17 17:50:24.150756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:24.150727 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8ed546-b55a-4c9a-b239-1b230ca267fa" path="/var/lib/kubelet/pods/3f8ed546-b55a-4c9a-b239-1b230ca267fa/volumes" Sep 17 17:50:24.151163 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:24.151105 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1da5568-6019-40c2-a850-452b1f0c2952" path="/var/lib/kubelet/pods/f1da5568-6019-40c2-a850-452b1f0c2952/volumes" Sep 17 17:50:28.373557 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.373469 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2"] Sep 17 17:50:28.374008 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.373726 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" podUID="d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" containerName="payload-processing" containerID="cri-o://f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f" gracePeriod=30 Sep 17 17:50:28.630740 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.630683 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:50:28.721501 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.721463 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-plugins-config-volume\") pod \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " Sep 17 17:50:28.721680 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.721554 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q9gz\" (UniqueName: \"kubernetes.io/projected/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-kube-api-access-7q9gz\") pod \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\" (UID: \"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc\") " Sep 17 17:50:28.721825 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.721800 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" (UID: "d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:50:28.723732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.723709 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-kube-api-access-7q9gz" (OuterVolumeSpecName: "kube-api-access-7q9gz") pod "d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" (UID: "d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc"). InnerVolumeSpecName "kube-api-access-7q9gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:50:28.822331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.822300 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:50:28.822331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:28.822326 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9gz\" (UniqueName: \"kubernetes.io/projected/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc-kube-api-access-7q9gz\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:50:29.360768 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.360733 2566 generic.go:356] "Generic (PLEG): container finished" podID="d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" containerID="f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f" exitCode=0 Sep 17 17:50:29.360942 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.360779 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" Sep 17 17:50:29.360942 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.360815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" event={"ID":"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc","Type":"ContainerDied","Data":"f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f"} Sep 17 17:50:29.360942 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.360872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2" event={"ID":"d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc","Type":"ContainerDied","Data":"2fd0a5544cc1aab5f38e17de908f4e21cbd6b9f83660c6ec3f2f83d010a5c2ac"} Sep 17 17:50:29.360942 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.360895 2566 scope.go:117] "RemoveContainer" containerID="f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f" Sep 17 17:50:29.370541 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.370522 2566 scope.go:117] "RemoveContainer" containerID="f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f" Sep 17 17:50:29.370870 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:50:29.370845 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f\": container with ID starting with f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f not found: ID does not exist" containerID="f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f" Sep 17 17:50:29.371007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.370878 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f"} err="failed to get container status \"f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f\": rpc error: code = NotFound desc = could not find container \"f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f\": container with ID starting with f0f1142db9fa21fde57532f648fdd9b098cbacfd68aa81c5dcc3d814e927149f not found: ID does not exist" Sep 17 17:50:29.386289 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.386267 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2"] Sep 17 17:50:29.394639 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:29.394620 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f6c1ea9-d694969bf-tf7g2"] Sep 17 17:50:30.150333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:50:30.150301 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" path="/var/lib/kubelet/pods/d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc/volumes" Sep 17 17:51:01.443083 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443044 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm"] Sep 17 17:51:01.443566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443486 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" containerName="payload-processing" Sep 17 17:51:01.443566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443498 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" containerName="payload-processing" Sep 17 17:51:01.443566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443508 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1da5568-6019-40c2-a850-452b1f0c2952" containerName="payload-pre-processing" Sep 17 17:51:01.443566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443513 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1da5568-6019-40c2-a850-452b1f0c2952" containerName="payload-pre-processing" Sep 17 17:51:01.443566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443529 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8ed546-b55a-4c9a-b239-1b230ca267fa" containerName="payload-processing" Sep 17 17:51:01.443566 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443534 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8ed546-b55a-4c9a-b239-1b230ca267fa" containerName="payload-processing" Sep 17 17:51:01.443790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443600 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1da5568-6019-40c2-a850-452b1f0c2952" containerName="payload-pre-processing" Sep 17 17:51:01.443790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443610 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8ed546-b55a-4c9a-b239-1b230ca267fa" containerName="payload-processing" Sep 17 17:51:01.443790 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.443618 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ffaf8c-3048-40b3-aab1-9eeaf0d393dc" containerName="payload-processing" Sep 17 17:51:01.446737 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.446713 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.449797 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.449773 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7e8755-dockercfg-6whm7\"" Sep 17 17:51:01.449914 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.449775 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7e8755\"" Sep 17 17:51:01.464304 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.464283 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm"] Sep 17 17:51:01.472371 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.472345 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp"] Sep 17 17:51:01.475727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.475709 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.495533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.495510 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp"] Sep 17 17:51:01.525899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.525862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ec3fbe-0ade-4297-ac79-a87bd7b589be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.526125 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.525928 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf88w\" (UniqueName: \"kubernetes.io/projected/87ec3fbe-0ade-4297-ac79-a87bd7b589be-kube-api-access-jf88w\") pod \"payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.626596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.626563 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ec3fbe-0ade-4297-ac79-a87bd7b589be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.626780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.626612 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf88w\" (UniqueName: \"kubernetes.io/projected/87ec3fbe-0ade-4297-ac79-a87bd7b589be-kube-api-access-jf88w\") pod \"payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.626780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.626644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjjj\" (UniqueName: \"kubernetes.io/projected/022026e5-e89e-487f-ba40-f04307e1365c-kube-api-access-dxjjj\") pod \"payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.626780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.626695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022026e5-e89e-487f-ba40-f04307e1365c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.627339 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.627321 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ec3fbe-0ade-4297-ac79-a87bd7b589be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.653703 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.653675 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf88w\" (UniqueName: \"kubernetes.io/projected/87ec3fbe-0ade-4297-ac79-a87bd7b589be-kube-api-access-jf88w\") pod \"payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:01.727299 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.727215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjjj\" (UniqueName: \"kubernetes.io/projected/022026e5-e89e-487f-ba40-f04307e1365c-kube-api-access-dxjjj\") pod \"payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.727457 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.727303 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022026e5-e89e-487f-ba40-f04307e1365c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.728029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.728011 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022026e5-e89e-487f-ba40-f04307e1365c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.745049 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.745027 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjjj\" (UniqueName: \"kubernetes.io/projected/022026e5-e89e-487f-ba40-f04307e1365c-kube-api-access-dxjjj\") pod \"payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.756892 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.756870 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:01.786197 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:01.786153 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:02.045338 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.045309 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm"] Sep 17 17:51:02.047567 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:51:02.047537 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022026e5_e89e_487f_ba40_f04307e1365c.slice/crio-a1c307a5755f8fa6fef87c5ac6333d3f6cdfcb468d085261b23d525048f02977 WatchSource:0}: Error finding container a1c307a5755f8fa6fef87c5ac6333d3f6cdfcb468d085261b23d525048f02977: Status 404 returned error can't find the container with id a1c307a5755f8fa6fef87c5ac6333d3f6cdfcb468d085261b23d525048f02977 Sep 17 17:51:02.076744 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.076667 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp"] Sep 17 17:51:02.084121 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:51:02.084097 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ec3fbe_0ade_4297_ac79_a87bd7b589be.slice/crio-12df58023a355c484e5e26fdf42e3fa059cb525aac78a4fd1ff79f9ca8ae9db1 WatchSource:0}: Error finding container 12df58023a355c484e5e26fdf42e3fa059cb525aac78a4fd1ff79f9ca8ae9db1: Status 404 returned error can't find the container with id 12df58023a355c484e5e26fdf42e3fa059cb525aac78a4fd1ff79f9ca8ae9db1 Sep 17 17:51:02.537932 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.537892 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" event={"ID":"87ec3fbe-0ade-4297-ac79-a87bd7b589be","Type":"ContainerStarted","Data":"33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2"} Sep 17 17:51:02.538432 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.537975 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:02.538432 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.538002 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" event={"ID":"87ec3fbe-0ade-4297-ac79-a87bd7b589be","Type":"ContainerStarted","Data":"12df58023a355c484e5e26fdf42e3fa059cb525aac78a4fd1ff79f9ca8ae9db1"} Sep 17 17:51:02.539734 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.539713 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" event={"ID":"022026e5-e89e-487f-ba40-f04307e1365c","Type":"ContainerStarted","Data":"33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5"} Sep 17 17:51:02.539873 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.539753 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" event={"ID":"022026e5-e89e-487f-ba40-f04307e1365c","Type":"ContainerStarted","Data":"a1c307a5755f8fa6fef87c5ac6333d3f6cdfcb468d085261b23d525048f02977"} Sep 17 17:51:02.539873 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.539782 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:02.557202 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.557157 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" podStartSLOduration=1.557141837 podStartE2EDuration="1.557141837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:02.554892181 +0000 UTC m=+1199.117057043" watchObservedRunningTime="2026-09-17 17:51:02.557141837 +0000 UTC m=+1199.119306696" Sep 17 17:51:02.574720 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:02.574662 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" podStartSLOduration=1.574647171 podStartE2EDuration="1.574647171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:02.573202765 +0000 UTC m=+1199.135367626" watchObservedRunningTime="2026-09-17 17:51:02.574647171 +0000 UTC m=+1199.136812075" Sep 17 17:51:13.547037 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:13.547002 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:51:13.547494 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:13.547190 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:51:22.169526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.169493 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7"] Sep 17 17:51:22.173816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.173797 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.176461 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.176437 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db7813\"/\"distinct-w0-db7813-kserve-self-signed-certs\"" Sep 17 17:51:22.185197 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.185176 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7"] Sep 17 17:51:22.302826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.302791 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tls-certs\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.302996 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.302834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mlv\" (UniqueName: \"kubernetes.io/projected/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-kube-api-access-z7mlv\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.302996 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.302855 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-dshm\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.302996 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.302966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-home\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.303112 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.303018 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tmp-dir\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.303112 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.303046 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-model-cache\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.403801 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.403766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mlv\" (UniqueName: \"kubernetes.io/projected/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-kube-api-access-z7mlv\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.403801 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.403801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-dshm\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.403833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-home\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.403855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tmp-dir\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.404006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-model-cache\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404237 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.404129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tls-certs\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404298 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.404254 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tmp-dir\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404298 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.404268 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-home\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.404405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.404345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-model-cache\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.406116 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.406094 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-dshm\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.406471 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.406452 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tls-certs\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.412585 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.412560 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mlv\" (UniqueName: \"kubernetes.io/projected/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-kube-api-access-z7mlv\") pod \"distinct-w0-db7813-kserve-85ddf7878b-nd2l7\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.483228 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.483156 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:22.759018 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:22.758987 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7"] Sep 17 17:51:22.760243 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:51:22.760217 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ebae666_aa3e_4d74_8f29_a94e997c5b3c.slice/crio-dd828780821b1e8e29bce47aff16a63eb9435f809f633cb63267cbccae7eff85 WatchSource:0}: Error finding container dd828780821b1e8e29bce47aff16a63eb9435f809f633cb63267cbccae7eff85: Status 404 returned error can't find the container with id dd828780821b1e8e29bce47aff16a63eb9435f809f633cb63267cbccae7eff85 Sep 17 17:51:23.652640 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:23.652603 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" event={"ID":"3ebae666-aa3e-4d74-8f29-a94e997c5b3c","Type":"ContainerStarted","Data":"bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89"} Sep 17 17:51:23.653137 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:23.652649 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" event={"ID":"3ebae666-aa3e-4d74-8f29-a94e997c5b3c","Type":"ContainerStarted","Data":"dd828780821b1e8e29bce47aff16a63eb9435f809f633cb63267cbccae7eff85"} Sep 17 17:51:23.686580 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:23.686529 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" podStartSLOduration=1.6865121090000001 podStartE2EDuration="1.686512109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:23.686228077 +0000 UTC m=+1220.248392951" watchObservedRunningTime="2026-09-17 17:51:23.686512109 +0000 UTC m=+1220.248676975" Sep 17 17:51:32.483877 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:32.483843 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:32.484320 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:32.483904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:32.496247 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:32.496229 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:32.712285 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:32.712257 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:51:38.967530 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:38.967476 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t"] Sep 17 17:51:38.973137 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:38.973111 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:38.978974 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:38.978752 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ef41f8-dockercfg-64xmk\"" Sep 17 17:51:38.978974 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:38.978941 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ef41f8\"" Sep 17 17:51:38.991686 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:38.991655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t"] Sep 17 17:51:39.065495 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.065452 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6h97\" (UniqueName: \"kubernetes.io/projected/c14c0eec-b53a-4df9-aac9-38a71f2f939c-kube-api-access-z6h97\") pod \"payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.065672 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.065576 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14c0eec-b53a-4df9-aac9-38a71f2f939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.169585 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.169549 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6h97\" (UniqueName: \"kubernetes.io/projected/c14c0eec-b53a-4df9-aac9-38a71f2f939c-kube-api-access-z6h97\") pod \"payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.169783 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.169638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14c0eec-b53a-4df9-aac9-38a71f2f939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.170684 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.170653 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14c0eec-b53a-4df9-aac9-38a71f2f939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.184571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.184544 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6h97\" (UniqueName: \"kubernetes.io/projected/c14c0eec-b53a-4df9-aac9-38a71f2f939c-kube-api-access-z6h97\") pod \"payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.296536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.296194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.599394 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.599365 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t"] Sep 17 17:51:39.601355 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:51:39.601326 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14c0eec_b53a_4df9_aac9_38a71f2f939c.slice/crio-84163ae91dff8921d4c434f73d01cddcda00dd3f491ac1784bddd5f70d87b179 WatchSource:0}: Error finding container 84163ae91dff8921d4c434f73d01cddcda00dd3f491ac1784bddd5f70d87b179: Status 404 returned error can't find the container with id 84163ae91dff8921d4c434f73d01cddcda00dd3f491ac1784bddd5f70d87b179 Sep 17 17:51:39.744249 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.744196 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" event={"ID":"c14c0eec-b53a-4df9-aac9-38a71f2f939c","Type":"ContainerStarted","Data":"f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee"} Sep 17 17:51:39.744249 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.744246 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" event={"ID":"c14c0eec-b53a-4df9-aac9-38a71f2f939c","Type":"ContainerStarted","Data":"84163ae91dff8921d4c434f73d01cddcda00dd3f491ac1784bddd5f70d87b179"} Sep 17 17:51:39.744504 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.744276 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:39.765676 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:39.765615 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" podStartSLOduration=1.765592849 podStartE2EDuration="1.765592849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:39.763127223 +0000 UTC m=+1236.325292087" watchObservedRunningTime="2026-09-17 17:51:39.765592849 +0000 UTC m=+1236.327757711" Sep 17 17:51:50.696092 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.696058 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6"] Sep 17 17:51:50.699528 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.699511 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.705512 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.705488 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-850d96\"/\"kube-root-ca.crt\"" Sep 17 17:51:50.706669 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.706649 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-850d96\"/\"openshift-service-ca.crt\"" Sep 17 17:51:50.706780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.706663 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-850d96\"/\"default-dockercfg-6gkqd\"" Sep 17 17:51:50.706780 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.706716 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-850d96\"/\"noten-model-850d96-kserve-self-signed-certs\"" Sep 17 17:51:50.719221 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.719196 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6"] Sep 17 17:51:50.752391 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.752368 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:51:50.783374 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.783339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51873828-1379-4077-9969-73930cdbf9f3-tls-certs\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.783526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.783378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-home\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.783526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.783461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-dshm\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.783526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.783477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbbst\" (UniqueName: \"kubernetes.io/projected/51873828-1379-4077-9969-73930cdbf9f3-kube-api-access-jbbst\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.783636 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.783536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-model-cache\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.783697 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.783675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-tmp-dir\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-dshm\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbst\" (UniqueName: \"kubernetes.io/projected/51873828-1379-4077-9969-73930cdbf9f3-kube-api-access-jbbst\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884240 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-model-cache\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-tmp-dir\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884338 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51873828-1379-4077-9969-73930cdbf9f3-tls-certs\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884367 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-home\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884661 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884629 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-model-cache\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884661 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884644 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-tmp-dir\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.884764 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.884697 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-home\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.886379 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.886362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-dshm\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.886802 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.886782 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51873828-1379-4077-9969-73930cdbf9f3-tls-certs\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:50.893687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:50.893663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbst\" (UniqueName: \"kubernetes.io/projected/51873828-1379-4077-9969-73930cdbf9f3-kube-api-access-jbbst\") pod \"noten-model-850d96-kserve-84b7fb6475-2dsd6\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:51.008735 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:51.008638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:51:51.294107 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:51.294032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6"] Sep 17 17:51:51.297360 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:51:51.297332 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51873828_1379_4077_9969_73930cdbf9f3.slice/crio-7873dc2170c1935218411dd528ccb5d39d4e0b20d68eaac0e2b1561c7fd70034 WatchSource:0}: Error finding container 7873dc2170c1935218411dd528ccb5d39d4e0b20d68eaac0e2b1561c7fd70034: Status 404 returned error can't find the container with id 7873dc2170c1935218411dd528ccb5d39d4e0b20d68eaac0e2b1561c7fd70034 Sep 17 17:51:51.813526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:51.813486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" event={"ID":"51873828-1379-4077-9969-73930cdbf9f3","Type":"ContainerStarted","Data":"6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306"} Sep 17 17:51:51.813526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:51.813529 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" event={"ID":"51873828-1379-4077-9969-73930cdbf9f3","Type":"ContainerStarted","Data":"7873dc2170c1935218411dd528ccb5d39d4e0b20d68eaac0e2b1561c7fd70034"} Sep 17 17:51:51.851266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:51:51.851222 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" podStartSLOduration=1.851207211 podStartE2EDuration="1.851207211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:51.849108307 +0000 UTC m=+1248.411273167" watchObservedRunningTime="2026-09-17 17:51:51.851207211 +0000 UTC m=+1248.413372111" Sep 17 17:52:01.009148 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:01.009063 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:52:01.009148 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:01.009117 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:52:01.021832 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:01.021801 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:52:01.875634 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:01.875606 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:52:02.658009 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.657976 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l"] Sep 17 17:52:02.658509 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.658265 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" podUID="543d4b92-7d49-4846-8b82-beb577a61dff" containerName="maas-api" containerID="cri-o://10e436589786d0d10f124c91849a3102097f59598e6a37acf8339510b51adea5" gracePeriod=30 Sep 17 17:52:02.768510 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.768465 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq"] Sep 17 17:52:02.769236 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.769206 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" podUID="1efb262c-45d6-46d5-b805-93af12aaca69" containerName="payload-processing" containerID="cri-o://4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996" gracePeriod=30 Sep 17 17:52:02.873266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.873222 2566 generic.go:356] "Generic (PLEG): container finished" podID="543d4b92-7d49-4846-8b82-beb577a61dff" containerID="10e436589786d0d10f124c91849a3102097f59598e6a37acf8339510b51adea5" exitCode=0 Sep 17 17:52:02.873512 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.873481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" event={"ID":"543d4b92-7d49-4846-8b82-beb577a61dff","Type":"ContainerDied","Data":"10e436589786d0d10f124c91849a3102097f59598e6a37acf8339510b51adea5"} Sep 17 17:52:02.927848 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.927785 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:52:02.996977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.996936 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtjkt\" (UniqueName: \"kubernetes.io/projected/543d4b92-7d49-4846-8b82-beb577a61dff-kube-api-access-gtjkt\") pod \"543d4b92-7d49-4846-8b82-beb577a61dff\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " Sep 17 17:52:02.997159 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.996996 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-maas-api-tls\") pod \"543d4b92-7d49-4846-8b82-beb577a61dff\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " Sep 17 17:52:02.997159 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.997111 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543d4b92-7d49-4846-8b82-beb577a61dff-openshift-service-ca\") pod \"543d4b92-7d49-4846-8b82-beb577a61dff\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " Sep 17 17:52:02.997253 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.997168 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-metrics-certs\") pod \"543d4b92-7d49-4846-8b82-beb577a61dff\" (UID: \"543d4b92-7d49-4846-8b82-beb577a61dff\") " Sep 17 17:52:02.997808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.997449 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543d4b92-7d49-4846-8b82-beb577a61dff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "543d4b92-7d49-4846-8b82-beb577a61dff" (UID: "543d4b92-7d49-4846-8b82-beb577a61dff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:52:02.997808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.997535 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543d4b92-7d49-4846-8b82-beb577a61dff-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:02.999254 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.999224 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "543d4b92-7d49-4846-8b82-beb577a61dff" (UID: "543d4b92-7d49-4846-8b82-beb577a61dff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:52:02.999351 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.999256 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "543d4b92-7d49-4846-8b82-beb577a61dff" (UID: "543d4b92-7d49-4846-8b82-beb577a61dff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:52:02.999351 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:02.999299 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543d4b92-7d49-4846-8b82-beb577a61dff-kube-api-access-gtjkt" (OuterVolumeSpecName: "kube-api-access-gtjkt") pod "543d4b92-7d49-4846-8b82-beb577a61dff" (UID: "543d4b92-7d49-4846-8b82-beb577a61dff"). InnerVolumeSpecName "kube-api-access-gtjkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:52:03.040482 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.040441 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:52:03.098817 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.098785 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhf28\" (UniqueName: \"kubernetes.io/projected/1efb262c-45d6-46d5-b805-93af12aaca69-kube-api-access-bhf28\") pod \"1efb262c-45d6-46d5-b805-93af12aaca69\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " Sep 17 17:52:03.098995 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.098891 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efb262c-45d6-46d5-b805-93af12aaca69-plugins-config-volume\") pod \"1efb262c-45d6-46d5-b805-93af12aaca69\" (UID: \"1efb262c-45d6-46d5-b805-93af12aaca69\") " Sep 17 17:52:03.099234 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.099205 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efb262c-45d6-46d5-b805-93af12aaca69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1efb262c-45d6-46d5-b805-93af12aaca69" (UID: "1efb262c-45d6-46d5-b805-93af12aaca69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:52:03.099234 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.099223 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtjkt\" (UniqueName: \"kubernetes.io/projected/543d4b92-7d49-4846-8b82-beb577a61dff-kube-api-access-gtjkt\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:03.099443 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.099242 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:03.099443 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.099251 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543d4b92-7d49-4846-8b82-beb577a61dff-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:03.100875 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.100857 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efb262c-45d6-46d5-b805-93af12aaca69-kube-api-access-bhf28" (OuterVolumeSpecName: "kube-api-access-bhf28") pod "1efb262c-45d6-46d5-b805-93af12aaca69" (UID: "1efb262c-45d6-46d5-b805-93af12aaca69"). InnerVolumeSpecName "kube-api-access-bhf28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:52:03.200049 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.199940 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhf28\" (UniqueName: \"kubernetes.io/projected/1efb262c-45d6-46d5-b805-93af12aaca69-kube-api-access-bhf28\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:03.200049 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.199994 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efb262c-45d6-46d5-b805-93af12aaca69-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:03.880891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.880859 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" Sep 17 17:52:03.880891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.880876 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l" event={"ID":"543d4b92-7d49-4846-8b82-beb577a61dff","Type":"ContainerDied","Data":"be501a2d802c5915df4c5ebab0771785f510bce1559e3e7c66071d4901c9daa7"} Sep 17 17:52:03.881410 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.880945 2566 scope.go:117] "RemoveContainer" containerID="10e436589786d0d10f124c91849a3102097f59598e6a37acf8339510b51adea5" Sep 17 17:52:03.882426 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.882402 2566 generic.go:356] "Generic (PLEG): container finished" podID="1efb262c-45d6-46d5-b805-93af12aaca69" containerID="4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996" exitCode=0 Sep 17 17:52:03.882486 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.882441 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" Sep 17 17:52:03.882546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.882482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" event={"ID":"1efb262c-45d6-46d5-b805-93af12aaca69","Type":"ContainerDied","Data":"4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996"} Sep 17 17:52:03.882546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.882533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq" event={"ID":"1efb262c-45d6-46d5-b805-93af12aaca69","Type":"ContainerDied","Data":"46a2ad31901d46d572303b70cf9591d31a74e1471d74ef34718484be8cf99da2"} Sep 17 17:52:03.890596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.890324 2566 scope.go:117] "RemoveContainer" containerID="4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996" Sep 17 17:52:03.901461 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.901438 2566 scope.go:117] "RemoveContainer" containerID="4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996" Sep 17 17:52:03.901741 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:03.901722 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996\": container with ID starting with 4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996 not found: ID does not exist" containerID="4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996" Sep 17 17:52:03.901811 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.901749 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996"} err="failed to get container status \"4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996\": rpc error: code = NotFound desc = could not find container \"4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996\": container with ID starting with 4ed3a78c405226f1bb3edb4bae10cca488291a018777317fb85022dbdc158996 not found: ID does not exist" Sep 17 17:52:03.912381 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.912359 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq"] Sep 17 17:52:03.916724 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.916703 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54966d-957c6685-8mdpq"] Sep 17 17:52:03.935609 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.935548 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l"] Sep 17 17:52:03.945060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:03.945040 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54966d-75bc6967f7-hxr9l"] Sep 17 17:52:04.152026 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:04.152000 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1efb262c-45d6-46d5-b805-93af12aaca69" path="/var/lib/kubelet/pods/1efb262c-45d6-46d5-b805-93af12aaca69/volumes" Sep 17 17:52:04.152392 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:04.152379 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543d4b92-7d49-4846-8b82-beb577a61dff" path="/var/lib/kubelet/pods/543d4b92-7d49-4846-8b82-beb577a61dff/volumes" Sep 17 17:52:06.858911 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.858879 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-575465"] Sep 17 17:52:06.859333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.859317 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543d4b92-7d49-4846-8b82-beb577a61dff" containerName="maas-api" Sep 17 17:52:06.859333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.859333 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="543d4b92-7d49-4846-8b82-beb577a61dff" containerName="maas-api" Sep 17 17:52:06.859409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.859344 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efb262c-45d6-46d5-b805-93af12aaca69" containerName="payload-processing" Sep 17 17:52:06.859409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.859350 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efb262c-45d6-46d5-b805-93af12aaca69" containerName="payload-processing" Sep 17 17:52:06.859491 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.859416 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efb262c-45d6-46d5-b805-93af12aaca69" containerName="payload-processing" Sep 17 17:52:06.859491 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.859428 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="543d4b92-7d49-4846-8b82-beb577a61dff" containerName="maas-api" Sep 17 17:52:06.863917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.863897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:06.866536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.866507 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z7xqm\"" Sep 17 17:52:06.873528 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.873502 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-575465"] Sep 17 17:52:06.937601 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:06.937568 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9kbj\" (UniqueName: \"kubernetes.io/projected/b60d92f2-5e3c-4599-bafa-175c2f138ba2-kube-api-access-s9kbj\") pod \"test-curl-7600-575465\" (UID: \"b60d92f2-5e3c-4599-bafa-175c2f138ba2\") " pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:07.039272 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.039236 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9kbj\" (UniqueName: \"kubernetes.io/projected/b60d92f2-5e3c-4599-bafa-175c2f138ba2-kube-api-access-s9kbj\") pod \"test-curl-7600-575465\" (UID: \"b60d92f2-5e3c-4599-bafa-175c2f138ba2\") " pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:07.048203 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.048181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9kbj\" (UniqueName: \"kubernetes.io/projected/b60d92f2-5e3c-4599-bafa-175c2f138ba2-kube-api-access-s9kbj\") pod \"test-curl-7600-575465\" (UID: \"b60d92f2-5e3c-4599-bafa-175c2f138ba2\") " pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:07.174430 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.174249 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:07.457016 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.456979 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-575465"] Sep 17 17:52:07.459494 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:52:07.459472 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb60d92f2_5e3c_4599_bafa_175c2f138ba2.slice/crio-06c07c1274791707b337d367fc8f7b95eff81d9e0a7fe88ffaab8e65ca2befae WatchSource:0}: Error finding container 06c07c1274791707b337d367fc8f7b95eff81d9e0a7fe88ffaab8e65ca2befae: Status 404 returned error can't find the container with id 06c07c1274791707b337d367fc8f7b95eff81d9e0a7fe88ffaab8e65ca2befae Sep 17 17:52:07.906407 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.906373 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-575465" event={"ID":"b60d92f2-5e3c-4599-bafa-175c2f138ba2","Type":"ContainerStarted","Data":"189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e"} Sep 17 17:52:07.906407 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.906413 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-575465" event={"ID":"b60d92f2-5e3c-4599-bafa-175c2f138ba2","Type":"ContainerStarted","Data":"06c07c1274791707b337d367fc8f7b95eff81d9e0a7fe88ffaab8e65ca2befae"} Sep 17 17:52:07.957216 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:07.957168 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7600-575465" podStartSLOduration=1.95715258 podStartE2EDuration="1.95715258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:07.955235583 +0000 UTC m=+1264.517400443" watchObservedRunningTime="2026-09-17 17:52:07.95715258 +0000 UTC m=+1264.519317439" Sep 17 17:52:08.748474 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.748391 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-575465"] Sep 17 17:52:08.758618 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.758588 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-575465"] Sep 17 17:52:08.879927 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.879886 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-3067a1"] Sep 17 17:52:08.880496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.880466 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" containerName="test-curl-7600-575465" Sep 17 17:52:08.880496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.880492 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" containerName="test-curl-7600-575465" Sep 17 17:52:08.880703 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.880580 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" containerName="test-curl-7600-575465" Sep 17 17:52:08.883765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.883750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:08.893307 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.893283 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-3067a1"] Sep 17 17:52:08.914857 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:08.914831 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7600-575465" container="test-curl-7600-575465" Sep 17 17:52:08.956558 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:08.956530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpltz\" (UniqueName: \"kubernetes.io/projected/278b4bc8-c8a9-4566-b34a-88cc06e07063-kube-api-access-bpltz\") pod \"test-curl-7600-3067a1\" (UID: \"278b4bc8-c8a9-4566-b34a-88cc06e07063\") " pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:09.057653 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.057627 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpltz\" (UniqueName: \"kubernetes.io/projected/278b4bc8-c8a9-4566-b34a-88cc06e07063-kube-api-access-bpltz\") pod \"test-curl-7600-3067a1\" (UID: \"278b4bc8-c8a9-4566-b34a-88cc06e07063\") " pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:09.080202 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.080177 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpltz\" (UniqueName: \"kubernetes.io/projected/278b4bc8-c8a9-4566-b34a-88cc06e07063-kube-api-access-bpltz\") pod \"test-curl-7600-3067a1\" (UID: \"278b4bc8-c8a9-4566-b34a-88cc06e07063\") " pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:09.194819 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.194789 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:09.490456 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.490429 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-3067a1"] Sep 17 17:52:09.492830 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:52:09.492799 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod278b4bc8_c8a9_4566_b34a_88cc06e07063.slice/crio-4c18fc6352ad1b0d296c974bc41830c051c4ad38515f92cb5b28b072d74b7153 WatchSource:0}: Error finding container 4c18fc6352ad1b0d296c974bc41830c051c4ad38515f92cb5b28b072d74b7153: Status 404 returned error can't find the container with id 4c18fc6352ad1b0d296c974bc41830c051c4ad38515f92cb5b28b072d74b7153 Sep 17 17:52:09.920826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.920785 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-3067a1" event={"ID":"278b4bc8-c8a9-4566-b34a-88cc06e07063","Type":"ContainerStarted","Data":"2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a"} Sep 17 17:52:09.921308 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.920829 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7600-575465" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" containerName="test-curl-7600-575465" containerID="cri-o://189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e" gracePeriod=2 Sep 17 17:52:09.921308 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:09.920831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-3067a1" event={"ID":"278b4bc8-c8a9-4566-b34a-88cc06e07063","Type":"ContainerStarted","Data":"4c18fc6352ad1b0d296c974bc41830c051c4ad38515f92cb5b28b072d74b7153"} Sep 17 17:52:09.928707 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:09.928668 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-575465\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:09.975747 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:09.975708 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-575465\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:10.727105 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:10.727052 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7600-3067a1" podStartSLOduration=2.727036519 podStartE2EDuration="2.727036519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:09.973672371 +0000 UTC m=+1266.535837231" watchObservedRunningTime="2026-09-17 17:52:10.727036519 +0000 UTC m=+1267.289201379" Sep 17 17:52:10.728444 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:10.728423 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-3067a1"] Sep 17 17:52:10.733579 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:10.733554 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-3067a1"] Sep 17 17:52:10.926325 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:10.926292 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7600-3067a1" container="test-curl-7600-3067a1" Sep 17 17:52:11.931858 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:11.931823 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7600-3067a1" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" containerName="test-curl-7600-3067a1" containerID="cri-o://2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a" gracePeriod=2 Sep 17 17:52:11.934737 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:11.934707 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-3067a1\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:12.105324 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.105298 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:12.107610 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:12.107585 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-3067a1\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:12.182016 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.181928 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9kbj\" (UniqueName: \"kubernetes.io/projected/b60d92f2-5e3c-4599-bafa-175c2f138ba2-kube-api-access-s9kbj\") pod \"b60d92f2-5e3c-4599-bafa-175c2f138ba2\" (UID: \"b60d92f2-5e3c-4599-bafa-175c2f138ba2\") " Sep 17 17:52:12.183968 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.183921 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60d92f2-5e3c-4599-bafa-175c2f138ba2-kube-api-access-s9kbj" (OuterVolumeSpecName: "kube-api-access-s9kbj") pod "b60d92f2-5e3c-4599-bafa-175c2f138ba2" (UID: "b60d92f2-5e3c-4599-bafa-175c2f138ba2"). InnerVolumeSpecName "kube-api-access-s9kbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:52:12.283037 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.283008 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9kbj\" (UniqueName: \"kubernetes.io/projected/b60d92f2-5e3c-4599-bafa-175c2f138ba2-kube-api-access-s9kbj\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:12.937732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.937701 2566 generic.go:356] "Generic (PLEG): container finished" podID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" containerID="189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e" exitCode=137 Sep 17 17:52:12.938154 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.937740 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-575465" Sep 17 17:52:12.938154 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.937789 2566 scope.go:117] "RemoveContainer" containerID="189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e" Sep 17 17:52:12.946915 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.946898 2566 scope.go:117] "RemoveContainer" containerID="189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e" Sep 17 17:52:12.947207 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:12.947189 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e\": container with ID starting with 189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e not found: ID does not exist" containerID="189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e" Sep 17 17:52:12.947263 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:12.947215 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e"} err="failed to get container status \"189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e\": rpc error: code = NotFound desc = could not find container \"189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e\": container with ID starting with 189af4a3e6f8dc3dea79281bc00b98b461367758c2d05e468e8d1509f3f4119e not found: ID does not exist" Sep 17 17:52:14.088090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.088066 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:14.150148 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.150120 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60d92f2-5e3c-4599-bafa-175c2f138ba2" path="/var/lib/kubelet/pods/b60d92f2-5e3c-4599-bafa-175c2f138ba2/volumes" Sep 17 17:52:14.203108 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.203055 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpltz\" (UniqueName: \"kubernetes.io/projected/278b4bc8-c8a9-4566-b34a-88cc06e07063-kube-api-access-bpltz\") pod \"278b4bc8-c8a9-4566-b34a-88cc06e07063\" (UID: \"278b4bc8-c8a9-4566-b34a-88cc06e07063\") " Sep 17 17:52:14.204993 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.204965 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278b4bc8-c8a9-4566-b34a-88cc06e07063-kube-api-access-bpltz" (OuterVolumeSpecName: "kube-api-access-bpltz") pod "278b4bc8-c8a9-4566-b34a-88cc06e07063" (UID: "278b4bc8-c8a9-4566-b34a-88cc06e07063"). InnerVolumeSpecName "kube-api-access-bpltz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:52:14.304595 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.304567 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpltz\" (UniqueName: \"kubernetes.io/projected/278b4bc8-c8a9-4566-b34a-88cc06e07063-kube-api-access-bpltz\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:14.956014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.955950 2566 generic.go:356] "Generic (PLEG): container finished" podID="278b4bc8-c8a9-4566-b34a-88cc06e07063" containerID="2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a" exitCode=137 Sep 17 17:52:14.956169 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.956048 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-3067a1" Sep 17 17:52:14.956169 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.956061 2566 scope.go:117] "RemoveContainer" containerID="2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a" Sep 17 17:52:14.965425 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.965407 2566 scope.go:117] "RemoveContainer" containerID="2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a" Sep 17 17:52:14.965709 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:14.965690 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a\": container with ID starting with 2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a not found: ID does not exist" containerID="2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a" Sep 17 17:52:14.965757 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:14.965715 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a"} err="failed to get container status \"2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a\": rpc error: code = NotFound desc = could not find container \"2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a\": container with ID starting with 2a7f0b78c09faaec321c08d98b97e3546f443a8d439592085a9c1a2792981a0a not found: ID does not exist" Sep 17 17:52:16.150676 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:16.150641 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" path="/var/lib/kubelet/pods/278b4bc8-c8a9-4566-b34a-88cc06e07063/volumes" Sep 17 17:52:22.473480 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.473440 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8"] Sep 17 17:52:22.474076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.474057 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" containerName="test-curl-7600-3067a1" Sep 17 17:52:22.474076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.474074 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" containerName="test-curl-7600-3067a1" Sep 17 17:52:22.474211 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.474153 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="278b4bc8-c8a9-4566-b34a-88cc06e07063" containerName="test-curl-7600-3067a1" Sep 17 17:52:22.478564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.478537 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.483398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.483372 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f9224635-serving-cert\"" Sep 17 17:52:22.492694 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.492673 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8"] Sep 17 17:52:22.579811 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.579781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.579996 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.579818 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ksmv\" (UniqueName: \"kubernetes.io/projected/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-kube-api-access-7ksmv\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.579996 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.579942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-metrics-certs\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.580077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.580028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-openshift-service-ca\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.680793 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.680756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-metrics-certs\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.680980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.680821 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-openshift-service-ca\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.680980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.680873 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.680980 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.680904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ksmv\" (UniqueName: \"kubernetes.io/projected/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-kube-api-access-7ksmv\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.681112 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:22.681030 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-serving-cert: secret "maas-api-e2e-mt-f9224635-serving-cert" not found Sep 17 17:52:22.681112 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:22.681094 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls podName:5d75a016-f56e-43f8-ab6e-c7123cf4bf83 nodeName:}" failed. No retries permitted until 2026-09-17 17:52:23.181077085 +0000 UTC m=+1279.743241922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls") pod "maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" (UID: "5d75a016-f56e-43f8-ab6e-c7123cf4bf83") : secret "maas-api-e2e-mt-f9224635-serving-cert" not found Sep 17 17:52:22.681479 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.681454 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-openshift-service-ca\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.683286 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.683259 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-metrics-certs\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:22.692878 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:22.692859 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ksmv\" (UniqueName: \"kubernetes.io/projected/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-kube-api-access-7ksmv\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:23.185104 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:23.185069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:23.187469 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:23.187437 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls\") pod \"maas-api-e2e-mt-f9224635-65fd44848b-pmcd8\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:23.389754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:23.389723 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:23.684934 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:23.684895 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8"] Sep 17 17:52:23.688277 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:52:23.688216 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d75a016_f56e_43f8_ab6e_c7123cf4bf83.slice/crio-57dc7bf0908140441906d95c336a7d7a4988b3e15a1fe084dd2a182ced5edea4 WatchSource:0}: Error finding container 57dc7bf0908140441906d95c336a7d7a4988b3e15a1fe084dd2a182ced5edea4: Status 404 returned error can't find the container with id 57dc7bf0908140441906d95c336a7d7a4988b3e15a1fe084dd2a182ced5edea4 Sep 17 17:52:23.689675 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:23.689652 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:52:24.002568 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:24.002485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" event={"ID":"5d75a016-f56e-43f8-ab6e-c7123cf4bf83","Type":"ContainerStarted","Data":"57dc7bf0908140441906d95c336a7d7a4988b3e15a1fe084dd2a182ced5edea4"} Sep 17 17:52:25.009272 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:25.009229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" event={"ID":"5d75a016-f56e-43f8-ab6e-c7123cf4bf83","Type":"ContainerStarted","Data":"5c107eecece72a42276a93bdd0e4d420e8d932603418d754ca1dcabd060bc0f9"} Sep 17 17:52:25.009667 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:25.009411 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:27.295247 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.295194 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" podStartSLOduration=4.838148881 podStartE2EDuration="5.295176991s" totalImagesPullingTime="457.02811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:25.078708851 +0000 UTC m=+1281.640873710" watchObservedRunningTime="2026-09-17 17:52:27.295176991 +0000 UTC m=+1283.857341851" Sep 17 17:52:27.295638 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.295333 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2"] Sep 17 17:52:27.295638 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.295510 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" podUID="ef15b92a-bda7-48a1-a2dd-029e3bb847f7" containerName="maas-api" containerID="cri-o://d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b" gracePeriod=30 Sep 17 17:52:27.569905 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.569880 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:52:27.624816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.624783 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") pod \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " Sep 17 17:52:27.624993 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.624844 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg7rm\" (UniqueName: \"kubernetes.io/projected/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-kube-api-access-dg7rm\") pod \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " Sep 17 17:52:27.624993 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.624938 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-openshift-service-ca\") pod \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " Sep 17 17:52:27.625089 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.625035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-metrics-certs\") pod \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\" (UID: \"ef15b92a-bda7-48a1-a2dd-029e3bb847f7\") " Sep 17 17:52:27.625264 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.625241 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef15b92a-bda7-48a1-a2dd-029e3bb847f7" (UID: "ef15b92a-bda7-48a1-a2dd-029e3bb847f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:52:27.627004 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.626977 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-kube-api-access-dg7rm" (OuterVolumeSpecName: "kube-api-access-dg7rm") pod "ef15b92a-bda7-48a1-a2dd-029e3bb847f7" (UID: "ef15b92a-bda7-48a1-a2dd-029e3bb847f7"). InnerVolumeSpecName "kube-api-access-dg7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:52:27.627120 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.627098 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef15b92a-bda7-48a1-a2dd-029e3bb847f7" (UID: "ef15b92a-bda7-48a1-a2dd-029e3bb847f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:52:27.627218 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.627173 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef15b92a-bda7-48a1-a2dd-029e3bb847f7" (UID: "ef15b92a-bda7-48a1-a2dd-029e3bb847f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:52:27.725765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.725736 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:27.725765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.725760 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:27.725765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.725771 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg7rm\" (UniqueName: \"kubernetes.io/projected/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-kube-api-access-dg7rm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:27.725950 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:27.725780 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef15b92a-bda7-48a1-a2dd-029e3bb847f7-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:28.026492 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.026460 2566 generic.go:356] "Generic (PLEG): container finished" podID="ef15b92a-bda7-48a1-a2dd-029e3bb847f7" containerID="d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b" exitCode=0 Sep 17 17:52:28.026672 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.026506 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" Sep 17 17:52:28.026672 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.026551 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" event={"ID":"ef15b92a-bda7-48a1-a2dd-029e3bb847f7","Type":"ContainerDied","Data":"d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b"} Sep 17 17:52:28.026672 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.026605 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2" event={"ID":"ef15b92a-bda7-48a1-a2dd-029e3bb847f7","Type":"ContainerDied","Data":"4ba6b9ddf94cb7d158f85dd50da3ae8f6dcaffc532a3c0ba8499d6f182625ca8"} Sep 17 17:52:28.026672 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.026629 2566 scope.go:117] "RemoveContainer" containerID="d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b" Sep 17 17:52:28.036413 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.036402 2566 scope.go:117] "RemoveContainer" containerID="d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b" Sep 17 17:52:28.036678 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:28.036644 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b\": container with ID starting with d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b not found: ID does not exist" containerID="d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b" Sep 17 17:52:28.036678 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.036671 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b"} err="failed to get container status \"d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b\": rpc error: code = NotFound desc = could not find container \"d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b\": container with ID starting with d56437cd1da5e14caf8d92c747c598bb8b688ffdf28fc00ff8d5d0b03fc7875b not found: ID does not exist" Sep 17 17:52:28.055251 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.055226 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2"] Sep 17 17:52:28.060379 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.060357 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfb277-5fcd9dd784-wl7x2"] Sep 17 17:52:28.126798 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.126770 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6"] Sep 17 17:52:28.151352 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:28.151324 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef15b92a-bda7-48a1-a2dd-029e3bb847f7" path="/var/lib/kubelet/pods/ef15b92a-bda7-48a1-a2dd-029e3bb847f7/volumes" Sep 17 17:52:31.020429 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:31.020402 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:52:42.744572 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.744540 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn"] Sep 17 17:52:42.745195 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.745164 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef15b92a-bda7-48a1-a2dd-029e3bb847f7" containerName="maas-api" Sep 17 17:52:42.745195 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.745190 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef15b92a-bda7-48a1-a2dd-029e3bb847f7" containerName="maas-api" Sep 17 17:52:42.745358 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.745279 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef15b92a-bda7-48a1-a2dd-029e3bb847f7" containerName="maas-api" Sep 17 17:52:42.748420 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.748397 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.753899 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.753878 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-abb363-serving-cert\"" Sep 17 17:52:42.771796 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.771770 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn"] Sep 17 17:52:42.869997 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.869943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.870162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.870017 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4lgj\" (UniqueName: \"kubernetes.io/projected/70e62bff-be8d-4718-ad4f-f32051fb281f-kube-api-access-p4lgj\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.870162 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.870072 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e62bff-be8d-4718-ad4f-f32051fb281f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.870264 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.870196 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.971290 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.971254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4lgj\" (UniqueName: \"kubernetes.io/projected/70e62bff-be8d-4718-ad4f-f32051fb281f-kube-api-access-p4lgj\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.971468 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.971316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e62bff-be8d-4718-ad4f-f32051fb281f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.971468 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.971378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.971468 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.971463 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.972134 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.972081 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e62bff-be8d-4718-ad4f-f32051fb281f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.974134 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.974108 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.974235 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.974214 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:42.990037 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:42.990012 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4lgj\" (UniqueName: \"kubernetes.io/projected/70e62bff-be8d-4718-ad4f-f32051fb281f-kube-api-access-p4lgj\") pod \"maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:43.059446 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.059421 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:43.145835 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.145795 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" podUID="51873828-1379-4077-9969-73930cdbf9f3" containerName="main" containerID="cri-o://6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306" gracePeriod=45 Sep 17 17:52:43.335987 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.335658 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn"] Sep 17 17:52:43.339526 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:52:43.339499 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e62bff_be8d_4718_ad4f_f32051fb281f.slice/crio-88b7de7ebcf842ee11ea9f623d669b4247278f372db30d63bfcea8e6f30b2759 WatchSource:0}: Error finding container 88b7de7ebcf842ee11ea9f623d669b4247278f372db30d63bfcea8e6f30b2759: Status 404 returned error can't find the container with id 88b7de7ebcf842ee11ea9f623d669b4247278f372db30d63bfcea8e6f30b2759 Sep 17 17:52:43.395607 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.395570 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:52:43.476738 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.476705 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-home\") pod \"51873828-1379-4077-9969-73930cdbf9f3\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " Sep 17 17:52:43.476738 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.476738 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51873828-1379-4077-9969-73930cdbf9f3-tls-certs\") pod \"51873828-1379-4077-9969-73930cdbf9f3\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " Sep 17 17:52:43.477007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.476768 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbbst\" (UniqueName: \"kubernetes.io/projected/51873828-1379-4077-9969-73930cdbf9f3-kube-api-access-jbbst\") pod \"51873828-1379-4077-9969-73930cdbf9f3\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " Sep 17 17:52:43.477007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.476839 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-tmp-dir\") pod \"51873828-1379-4077-9969-73930cdbf9f3\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " Sep 17 17:52:43.477007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.476872 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-dshm\") pod \"51873828-1379-4077-9969-73930cdbf9f3\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " Sep 17 17:52:43.477007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.476894 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-model-cache\") pod \"51873828-1379-4077-9969-73930cdbf9f3\" (UID: \"51873828-1379-4077-9969-73930cdbf9f3\") " Sep 17 17:52:43.477219 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.477041 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-home" (OuterVolumeSpecName: "home") pod "51873828-1379-4077-9969-73930cdbf9f3" (UID: "51873828-1379-4077-9969-73930cdbf9f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:52:43.477219 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.477129 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51873828-1379-4077-9969-73930cdbf9f3" (UID: "51873828-1379-4077-9969-73930cdbf9f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:52:43.477360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.477336 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "51873828-1379-4077-9969-73930cdbf9f3" (UID: "51873828-1379-4077-9969-73930cdbf9f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:52:43.477442 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.477421 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:43.477497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.477448 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:43.477497 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.477462 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:43.479081 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.479051 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51873828-1379-4077-9969-73930cdbf9f3-kube-api-access-jbbst" (OuterVolumeSpecName: "kube-api-access-jbbst") pod "51873828-1379-4077-9969-73930cdbf9f3" (UID: "51873828-1379-4077-9969-73930cdbf9f3"). InnerVolumeSpecName "kube-api-access-jbbst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:52:43.479207 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.479120 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-dshm" (OuterVolumeSpecName: "dshm") pod "51873828-1379-4077-9969-73930cdbf9f3" (UID: "51873828-1379-4077-9969-73930cdbf9f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:52:43.479254 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.479195 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51873828-1379-4077-9969-73930cdbf9f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51873828-1379-4077-9969-73930cdbf9f3" (UID: "51873828-1379-4077-9969-73930cdbf9f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:52:43.578748 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.578670 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51873828-1379-4077-9969-73930cdbf9f3-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:43.578748 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.578696 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbbst\" (UniqueName: \"kubernetes.io/projected/51873828-1379-4077-9969-73930cdbf9f3-kube-api-access-jbbst\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:43.578748 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:43.578705 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51873828-1379-4077-9969-73930cdbf9f3-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:52:44.118917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.118881 2566 generic.go:356] "Generic (PLEG): container finished" podID="51873828-1379-4077-9969-73930cdbf9f3" containerID="6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306" exitCode=0 Sep 17 17:52:44.118917 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.118919 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" Sep 17 17:52:44.119405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.118986 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" event={"ID":"51873828-1379-4077-9969-73930cdbf9f3","Type":"ContainerDied","Data":"6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306"} Sep 17 17:52:44.119405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.119029 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6" event={"ID":"51873828-1379-4077-9969-73930cdbf9f3","Type":"ContainerDied","Data":"7873dc2170c1935218411dd528ccb5d39d4e0b20d68eaac0e2b1561c7fd70034"} Sep 17 17:52:44.119405 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.119050 2566 scope.go:117] "RemoveContainer" containerID="6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306" Sep 17 17:52:44.120692 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.120666 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" event={"ID":"70e62bff-be8d-4718-ad4f-f32051fb281f","Type":"ContainerStarted","Data":"929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b"} Sep 17 17:52:44.120789 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.120717 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" event={"ID":"70e62bff-be8d-4718-ad4f-f32051fb281f","Type":"ContainerStarted","Data":"88b7de7ebcf842ee11ea9f623d669b4247278f372db30d63bfcea8e6f30b2759"} Sep 17 17:52:44.120842 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.120806 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:44.127942 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.127927 2566 scope.go:117] "RemoveContainer" containerID="6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306" Sep 17 17:52:44.128223 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:52:44.128202 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306\": container with ID starting with 6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306 not found: ID does not exist" containerID="6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306" Sep 17 17:52:44.128271 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.128231 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306"} err="failed to get container status \"6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306\": rpc error: code = NotFound desc = could not find container \"6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306\": container with ID starting with 6c1f025f332d4ccc10852923d96c63ff1d2fc87b291e9c50f63f4e0322c2a306 not found: ID does not exist" Sep 17 17:52:44.165867 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.165826 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" podStartSLOduration=1.75079065 podStartE2EDuration="2.165812057s" totalImagesPullingTime="415.021407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:44.162649226 +0000 UTC m=+1300.724814086" watchObservedRunningTime="2026-09-17 17:52:44.165812057 +0000 UTC m=+1300.727976917" Sep 17 17:52:44.215007 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.214981 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6"] Sep 17 17:52:44.225400 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:44.225376 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-850d96/noten-model-850d96-kserve-84b7fb6475-2dsd6"] Sep 17 17:52:46.158297 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:46.158259 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51873828-1379-4077-9969-73930cdbf9f3" path="/var/lib/kubelet/pods/51873828-1379-4077-9969-73930cdbf9f3/volumes" Sep 17 17:52:50.133222 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:50.133191 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 17:52:58.113833 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.113802 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr"] Sep 17 17:52:58.114223 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.114210 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51873828-1379-4077-9969-73930cdbf9f3" containerName="main" Sep 17 17:52:58.114223 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.114224 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51873828-1379-4077-9969-73930cdbf9f3" containerName="main" Sep 17 17:52:58.114322 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.114313 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51873828-1379-4077-9969-73930cdbf9f3" containerName="main" Sep 17 17:52:58.118543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.118523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.121043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.121018 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"e2e-conflict-b-42c1d347-kserve-self-signed-certs\"" Sep 17 17:52:58.126754 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.126730 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr"] Sep 17 17:52:58.219510 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.219479 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-model-cache\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.219682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.219557 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-home\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.219682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.219592 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-dshm\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.219682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.219629 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce77629-2a33-428f-8276-58057ecf432e-tls-certs\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.219814 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.219698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-tmp-dir\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.219814 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.219729 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvxp\" (UniqueName: \"kubernetes.io/projected/3ce77629-2a33-428f-8276-58057ecf432e-kube-api-access-snvxp\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.320829 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.320797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-home\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.320847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-dshm\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.320875 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce77629-2a33-428f-8276-58057ecf432e-tls-certs\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.320909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-tmp-dir\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321032 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.320928 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvxp\" (UniqueName: \"kubernetes.io/projected/3ce77629-2a33-428f-8276-58057ecf432e-kube-api-access-snvxp\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321299 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.321069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-model-cache\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321363 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.321341 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-home\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321425 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.321378 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-tmp-dir\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.321535 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.321513 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-model-cache\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.323578 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.323551 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-dshm\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.323812 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.323795 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce77629-2a33-428f-8276-58057ecf432e-tls-certs\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.329882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.329861 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvxp\" (UniqueName: \"kubernetes.io/projected/3ce77629-2a33-428f-8276-58057ecf432e-kube-api-access-snvxp\") pod \"e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.428189 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.428103 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:52:58.704109 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:58.704081 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr"] Sep 17 17:52:58.705746 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:52:58.705712 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ce77629_2a33_428f_8276_58057ecf432e.slice/crio-bfafaaabc84529e758ed7dbe080a06b6cb871de3e7fe5292952062069b2ace16 WatchSource:0}: Error finding container bfafaaabc84529e758ed7dbe080a06b6cb871de3e7fe5292952062069b2ace16: Status 404 returned error can't find the container with id bfafaaabc84529e758ed7dbe080a06b6cb871de3e7fe5292952062069b2ace16 Sep 17 17:52:59.199921 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:59.199880 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" event={"ID":"3ce77629-2a33-428f-8276-58057ecf432e","Type":"ContainerStarted","Data":"67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86"} Sep 17 17:52:59.200321 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:59.199929 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" event={"ID":"3ce77629-2a33-428f-8276-58057ecf432e","Type":"ContainerStarted","Data":"bfafaaabc84529e758ed7dbe080a06b6cb871de3e7fe5292952062069b2ace16"} Sep 17 17:52:59.222765 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:52:59.222722 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" podStartSLOduration=1.222709003 podStartE2EDuration="1.222709003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:59.221063815 +0000 UTC m=+1315.783228675" watchObservedRunningTime="2026-09-17 17:52:59.222709003 +0000 UTC m=+1315.784873863" Sep 17 17:53:08.429096 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:08.429057 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:53:08.429477 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:08.429112 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:53:08.442047 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:08.442016 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:53:09.263066 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:09.263035 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:53:14.075895 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.075854 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d"] Sep 17 17:53:14.081494 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.081469 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.084862 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.084828 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db7813\"/\"distinct-2-w0-db7813-kserve-self-signed-certs\"" Sep 17 17:53:14.101191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.101168 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d"] Sep 17 17:53:14.173573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.173539 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-dshm\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.173739 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.173607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vq6k\" (UniqueName: \"kubernetes.io/projected/fa80eefc-6844-47e2-ada0-9997376371c9-kube-api-access-6vq6k\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.173739 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.173656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-model-cache\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.173826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.173748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-tmp-dir\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.173826 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.173800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-home\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.173910 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.173874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa80eefc-6844-47e2-ada0-9997376371c9-tls-certs\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.274690 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.274655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa80eefc-6844-47e2-ada0-9997376371c9-tls-certs\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.274913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.274709 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-dshm\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.274913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.274747 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vq6k\" (UniqueName: \"kubernetes.io/projected/fa80eefc-6844-47e2-ada0-9997376371c9-kube-api-access-6vq6k\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.274913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.274775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-model-cache\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.274913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.274809 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-tmp-dir\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.275170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.274939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-home\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.275310 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.275288 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-tmp-dir\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.275379 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.275314 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-home\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.275517 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.275494 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-model-cache\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.277284 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.277256 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-dshm\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.277788 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.277768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa80eefc-6844-47e2-ada0-9997376371c9-tls-certs\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.289103 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.289076 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vq6k\" (UniqueName: \"kubernetes.io/projected/fa80eefc-6844-47e2-ada0-9997376371c9-kube-api-access-6vq6k\") pod \"distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.392191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.392111 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:14.671285 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:14.671255 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d"] Sep 17 17:53:14.673358 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:53:14.673315 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa80eefc_6844_47e2_ada0_9997376371c9.slice/crio-fd4b2b4fb3053dba0d7fe64dc896863db71776fb709f11334aa291d1a62ba0d8 WatchSource:0}: Error finding container fd4b2b4fb3053dba0d7fe64dc896863db71776fb709f11334aa291d1a62ba0d8: Status 404 returned error can't find the container with id fd4b2b4fb3053dba0d7fe64dc896863db71776fb709f11334aa291d1a62ba0d8 Sep 17 17:53:15.287872 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:15.287826 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" event={"ID":"fa80eefc-6844-47e2-ada0-9997376371c9","Type":"ContainerStarted","Data":"db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7"} Sep 17 17:53:15.288295 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:15.287880 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" event={"ID":"fa80eefc-6844-47e2-ada0-9997376371c9","Type":"ContainerStarted","Data":"fd4b2b4fb3053dba0d7fe64dc896863db71776fb709f11334aa291d1a62ba0d8"} Sep 17 17:53:15.328936 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:15.328892 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" podStartSLOduration=1.328877966 podStartE2EDuration="1.328877966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:15.327754627 +0000 UTC m=+1331.889919484" watchObservedRunningTime="2026-09-17 17:53:15.328877966 +0000 UTC m=+1331.891042826" Sep 17 17:53:24.393185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:24.393146 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:24.393681 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:24.393201 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:24.405699 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:24.405665 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:53:25.350087 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:53:25.350058 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:54:05.967070 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.967034 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s"] Sep 17 17:54:05.980166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.980137 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:05.982902 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.982877 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"auth-test-model-abb363-kserve-self-signed-certs\"" Sep 17 17:54:05.984035 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.984014 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"openshift-service-ca.crt\"" Sep 17 17:54:05.984158 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.984096 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"kube-root-ca.crt\"" Sep 17 17:54:05.984260 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.984236 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"default-dockercfg-k9lpm\"" Sep 17 17:54:05.987827 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:05.987662 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s"] Sep 17 17:54:06.059727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.059699 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-model-cache\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.059882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.059733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rmr\" (UniqueName: \"kubernetes.io/projected/66fb4f1d-067f-490c-8eda-734c9e7bcb37-kube-api-access-29rmr\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.059882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.059753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tls-certs\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.059882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.059826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-dshm\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.059882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.059840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-home\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.059882 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.059865 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tmp-dir\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.160850 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.160817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-dshm\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.160850 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.160849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-home\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.160871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tmp-dir\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.160900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-model-cache\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.160920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rmr\" (UniqueName: \"kubernetes.io/projected/66fb4f1d-067f-490c-8eda-734c9e7bcb37-kube-api-access-29rmr\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161098 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.160948 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tls-certs\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161358 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.161337 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-home\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.161377 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-model-cache\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.161424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.161378 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tmp-dir\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.163149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.163126 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-dshm\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.163353 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.163334 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tls-certs\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.170858 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.170833 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rmr\" (UniqueName: \"kubernetes.io/projected/66fb4f1d-067f-490c-8eda-734c9e7bcb37-kube-api-access-29rmr\") pod \"auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.290331 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.290253 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:06.573350 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:06.573318 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s"] Sep 17 17:54:06.576511 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:54:06.576480 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66fb4f1d_067f_490c_8eda_734c9e7bcb37.slice/crio-7c9ecb1d8e30045a1527bfcd9656a0926d4af0b46bec1b6da4ec406b28944f1b WatchSource:0}: Error finding container 7c9ecb1d8e30045a1527bfcd9656a0926d4af0b46bec1b6da4ec406b28944f1b: Status 404 returned error can't find the container with id 7c9ecb1d8e30045a1527bfcd9656a0926d4af0b46bec1b6da4ec406b28944f1b Sep 17 17:54:07.558810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:07.558779 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" event={"ID":"66fb4f1d-067f-490c-8eda-734c9e7bcb37","Type":"ContainerStarted","Data":"586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48"} Sep 17 17:54:07.559235 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:07.558818 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" event={"ID":"66fb4f1d-067f-490c-8eda-734c9e7bcb37","Type":"ContainerStarted","Data":"7c9ecb1d8e30045a1527bfcd9656a0926d4af0b46bec1b6da4ec406b28944f1b"} Sep 17 17:54:07.589660 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:07.589612 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" podStartSLOduration=2.589595235 podStartE2EDuration="2.589595235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:07.588254456 +0000 UTC m=+1384.150419317" watchObservedRunningTime="2026-09-17 17:54:07.589595235 +0000 UTC m=+1384.151760095" Sep 17 17:54:15.341198 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:15.341165 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr"] Sep 17 17:54:16.290552 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:16.290516 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:16.290727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:16.290563 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:16.302822 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:16.302787 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:16.616012 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:16.615983 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:54:30.360352 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.360287 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" podUID="3ce77629-2a33-428f-8276-58057ecf432e" containerName="main" containerID="cri-o://67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86" gracePeriod=45 Sep 17 17:54:30.625981 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.625945 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:54:30.679234 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.679198 2566 generic.go:356] "Generic (PLEG): container finished" podID="3ce77629-2a33-428f-8276-58057ecf432e" containerID="67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86" exitCode=0 Sep 17 17:54:30.679427 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.679240 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" Sep 17 17:54:30.679427 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.679285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" event={"ID":"3ce77629-2a33-428f-8276-58057ecf432e","Type":"ContainerDied","Data":"67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86"} Sep 17 17:54:30.679427 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.679346 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr" event={"ID":"3ce77629-2a33-428f-8276-58057ecf432e","Type":"ContainerDied","Data":"bfafaaabc84529e758ed7dbe080a06b6cb871de3e7fe5292952062069b2ace16"} Sep 17 17:54:30.679427 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.679369 2566 scope.go:117] "RemoveContainer" containerID="67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86" Sep 17 17:54:30.688167 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.688149 2566 scope.go:117] "RemoveContainer" containerID="67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86" Sep 17 17:54:30.688452 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:54:30.688432 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86\": container with ID starting with 67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86 not found: ID does not exist" containerID="67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86" Sep 17 17:54:30.688500 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.688462 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86"} err="failed to get container status \"67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86\": rpc error: code = NotFound desc = could not find container \"67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86\": container with ID starting with 67a6613ee9878794e4b1096f797d2d7ae9fd45a09324ea2b2d01cd9248122a86 not found: ID does not exist" Sep 17 17:54:30.801913 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.801876 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-dshm\") pod \"3ce77629-2a33-428f-8276-58057ecf432e\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " Sep 17 17:54:30.802127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.801944 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce77629-2a33-428f-8276-58057ecf432e-tls-certs\") pod \"3ce77629-2a33-428f-8276-58057ecf432e\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " Sep 17 17:54:30.802127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802019 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-model-cache\") pod \"3ce77629-2a33-428f-8276-58057ecf432e\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " Sep 17 17:54:30.802127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802044 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-tmp-dir\") pod \"3ce77629-2a33-428f-8276-58057ecf432e\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " Sep 17 17:54:30.802127 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802079 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-home\") pod \"3ce77629-2a33-428f-8276-58057ecf432e\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " Sep 17 17:54:30.802359 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802151 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snvxp\" (UniqueName: \"kubernetes.io/projected/3ce77629-2a33-428f-8276-58057ecf432e-kube-api-access-snvxp\") pod \"3ce77629-2a33-428f-8276-58057ecf432e\" (UID: \"3ce77629-2a33-428f-8276-58057ecf432e\") " Sep 17 17:54:30.802359 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802310 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ce77629-2a33-428f-8276-58057ecf432e" (UID: "3ce77629-2a33-428f-8276-58057ecf432e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:54:30.802359 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802342 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ce77629-2a33-428f-8276-58057ecf432e" (UID: "3ce77629-2a33-428f-8276-58057ecf432e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:54:30.802514 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802397 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-home" (OuterVolumeSpecName: "home") pod "3ce77629-2a33-428f-8276-58057ecf432e" (UID: "3ce77629-2a33-428f-8276-58057ecf432e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:54:30.802594 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802575 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:30.802650 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802603 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:30.802650 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.802616 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:30.804169 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.804147 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-dshm" (OuterVolumeSpecName: "dshm") pod "3ce77629-2a33-428f-8276-58057ecf432e" (UID: "3ce77629-2a33-428f-8276-58057ecf432e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:54:30.804266 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.804185 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce77629-2a33-428f-8276-58057ecf432e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ce77629-2a33-428f-8276-58057ecf432e" (UID: "3ce77629-2a33-428f-8276-58057ecf432e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:54:30.804305 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.804271 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce77629-2a33-428f-8276-58057ecf432e-kube-api-access-snvxp" (OuterVolumeSpecName: "kube-api-access-snvxp") pod "3ce77629-2a33-428f-8276-58057ecf432e" (UID: "3ce77629-2a33-428f-8276-58057ecf432e"). InnerVolumeSpecName "kube-api-access-snvxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:54:30.903722 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.903650 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvxp\" (UniqueName: \"kubernetes.io/projected/3ce77629-2a33-428f-8276-58057ecf432e-kube-api-access-snvxp\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:30.903722 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.903674 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce77629-2a33-428f-8276-58057ecf432e-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:30.903722 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:30.903683 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce77629-2a33-428f-8276-58057ecf432e-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:31.005570 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:31.005536 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr"] Sep 17 17:54:31.015249 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:31.015226 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/e2e-conflict-b-42c1d347-kserve-75cd574677-jjzrr"] Sep 17 17:54:32.150720 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:32.150686 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce77629-2a33-428f-8276-58057ecf432e" path="/var/lib/kubelet/pods/3ce77629-2a33-428f-8276-58057ecf432e/volumes" Sep 17 17:54:33.436509 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.436479 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7"] Sep 17 17:54:33.437021 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.436966 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce77629-2a33-428f-8276-58057ecf432e" containerName="main" Sep 17 17:54:33.437021 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.436997 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce77629-2a33-428f-8276-58057ecf432e" containerName="main" Sep 17 17:54:33.437242 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.437136 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce77629-2a33-428f-8276-58057ecf432e" containerName="main" Sep 17 17:54:33.441549 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.441527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.444639 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.444619 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8aeea1\"/\"distinct-w4-8aeea1-kserve-self-signed-certs\"" Sep 17 17:54:33.461723 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.461692 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7"] Sep 17 17:54:33.532977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.529946 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-model-cache\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.532977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.530061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tls-certs\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.532977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.530109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-home\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.532977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.530149 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tmp-dir\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.532977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.530199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-dshm\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.532977 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.530231 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbg6m\" (UniqueName: \"kubernetes.io/projected/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-kube-api-access-tbg6m\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631147 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-model-cache\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631317 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tls-certs\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631317 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631199 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-home\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631317 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tmp-dir\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631317 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631270 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-dshm\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631555 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbg6m\" (UniqueName: \"kubernetes.io/projected/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-kube-api-access-tbg6m\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631575 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-model-cache\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631669 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631618 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-home\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.631669 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.631635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tmp-dir\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.633466 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.633447 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-dshm\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.633812 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.633791 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tls-certs\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.641744 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.641726 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbg6m\" (UniqueName: \"kubernetes.io/projected/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-kube-api-access-tbg6m\") pod \"distinct-w4-8aeea1-kserve-955f7bc76-d5rl7\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:33.751026 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:33.750934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:34.039508 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:34.039476 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7"] Sep 17 17:54:34.040974 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:54:34.040922 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e7f93e_fe69_4a77_87d2_42a5154bd2ac.slice/crio-d7d18c72b6271c6662840ccef30b5bc2c740eb99fc34c122d737dfaa12ff4dd7 WatchSource:0}: Error finding container d7d18c72b6271c6662840ccef30b5bc2c740eb99fc34c122d737dfaa12ff4dd7: Status 404 returned error can't find the container with id d7d18c72b6271c6662840ccef30b5bc2c740eb99fc34c122d737dfaa12ff4dd7 Sep 17 17:54:34.703278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:34.703240 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" event={"ID":"52e7f93e-fe69-4a77-87d2-42a5154bd2ac","Type":"ContainerStarted","Data":"f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f"} Sep 17 17:54:34.703668 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:34.703286 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" event={"ID":"52e7f93e-fe69-4a77-87d2-42a5154bd2ac","Type":"ContainerStarted","Data":"d7d18c72b6271c6662840ccef30b5bc2c740eb99fc34c122d737dfaa12ff4dd7"} Sep 17 17:54:34.749392 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:34.749345 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" podStartSLOduration=1.749330989 podStartE2EDuration="1.749330989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:34.746509725 +0000 UTC m=+1411.308674586" watchObservedRunningTime="2026-09-17 17:54:34.749330989 +0000 UTC m=+1411.311495849" Sep 17 17:54:43.751386 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:43.751341 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:43.751827 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:43.751397 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:43.763847 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:43.763821 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:44.767691 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:44.767665 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:54:54.604753 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.604712 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8"] Sep 17 17:54:54.605268 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.604914 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" podUID="5d75a016-f56e-43f8-ab6e-c7123cf4bf83" containerName="maas-api" containerID="cri-o://5c107eecece72a42276a93bdd0e4d420e8d932603418d754ca1dcabd060bc0f9" gracePeriod=30 Sep 17 17:54:54.813511 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.813480 2566 generic.go:356] "Generic (PLEG): container finished" podID="5d75a016-f56e-43f8-ab6e-c7123cf4bf83" containerID="5c107eecece72a42276a93bdd0e4d420e8d932603418d754ca1dcabd060bc0f9" exitCode=0 Sep 17 17:54:54.813680 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.813557 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" event={"ID":"5d75a016-f56e-43f8-ab6e-c7123cf4bf83","Type":"ContainerDied","Data":"5c107eecece72a42276a93bdd0e4d420e8d932603418d754ca1dcabd060bc0f9"} Sep 17 17:54:54.863215 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.863160 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:54:54.938270 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.938241 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-openshift-service-ca\") pod \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " Sep 17 17:54:54.938422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.938283 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-metrics-certs\") pod \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " Sep 17 17:54:54.938422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.938320 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ksmv\" (UniqueName: \"kubernetes.io/projected/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-kube-api-access-7ksmv\") pod \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " Sep 17 17:54:54.938422 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.938365 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls\") pod \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\" (UID: \"5d75a016-f56e-43f8-ab6e-c7123cf4bf83\") " Sep 17 17:54:54.938596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.938568 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d75a016-f56e-43f8-ab6e-c7123cf4bf83" (UID: "5d75a016-f56e-43f8-ab6e-c7123cf4bf83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:54:54.940409 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.940385 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d75a016-f56e-43f8-ab6e-c7123cf4bf83" (UID: "5d75a016-f56e-43f8-ab6e-c7123cf4bf83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:54:54.940536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.940489 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-kube-api-access-7ksmv" (OuterVolumeSpecName: "kube-api-access-7ksmv") pod "5d75a016-f56e-43f8-ab6e-c7123cf4bf83" (UID: "5d75a016-f56e-43f8-ab6e-c7123cf4bf83"). InnerVolumeSpecName "kube-api-access-7ksmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:54:54.940536 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:54.940502 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5d75a016-f56e-43f8-ab6e-c7123cf4bf83" (UID: "5d75a016-f56e-43f8-ab6e-c7123cf4bf83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:54:55.040025 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.039989 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:55.040025 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.040019 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:55.040025 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.040029 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ksmv\" (UniqueName: \"kubernetes.io/projected/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-kube-api-access-7ksmv\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:55.040256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.040038 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d75a016-f56e-43f8-ab6e-c7123cf4bf83-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:54:55.820348 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.820321 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" Sep 17 17:54:55.820788 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.820353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8" event={"ID":"5d75a016-f56e-43f8-ab6e-c7123cf4bf83","Type":"ContainerDied","Data":"57dc7bf0908140441906d95c336a7d7a4988b3e15a1fe084dd2a182ced5edea4"} Sep 17 17:54:55.820788 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.820409 2566 scope.go:117] "RemoveContainer" containerID="5c107eecece72a42276a93bdd0e4d420e8d932603418d754ca1dcabd060bc0f9" Sep 17 17:54:55.852475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.852449 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8"] Sep 17 17:54:55.860617 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:55.860596 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9224635-65fd44848b-pmcd8"] Sep 17 17:54:56.151191 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:54:56.151097 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d75a016-f56e-43f8-ab6e-c7123cf4bf83" path="/var/lib/kubelet/pods/5d75a016-f56e-43f8-ab6e-c7123cf4bf83/volumes" Sep 17 17:55:10.418391 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.418357 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5"] Sep 17 17:55:10.419090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.419069 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d75a016-f56e-43f8-ab6e-c7123cf4bf83" containerName="maas-api" Sep 17 17:55:10.419090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.419090 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d75a016-f56e-43f8-ab6e-c7123cf4bf83" containerName="maas-api" Sep 17 17:55:10.419256 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.419222 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d75a016-f56e-43f8-ab6e-c7123cf4bf83" containerName="maas-api" Sep 17 17:55:10.423771 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.423752 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.426407 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.426377 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"default-dockercfg-n6kdw\"" Sep 17 17:55:10.426511 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.426416 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"openshift-service-ca.crt\"" Sep 17 17:55:10.427533 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.427513 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"auth-test-model-7b7c8c-kserve-self-signed-certs\"" Sep 17 17:55:10.427617 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.427519 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"kube-root-ca.crt\"" Sep 17 17:55:10.434879 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.434851 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5"] Sep 17 17:55:10.601512 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.601471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tmp-dir\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.601711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.601532 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jtl\" (UniqueName: \"kubernetes.io/projected/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-kube-api-access-r5jtl\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.601711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.601581 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-home\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.601711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.601599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-dshm\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.601711 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.601625 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-model-cache\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.601859 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.601733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tls-certs\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.702894 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.702800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tls-certs\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.702894 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.702846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tmp-dir\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.702899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jtl\" (UniqueName: \"kubernetes.io/projected/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-kube-api-access-r5jtl\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.703006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-home\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.703035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-dshm\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.703080 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-model-cache\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.703192 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tmp-dir\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.703294 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-home\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.703475 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.703447 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-model-cache\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.705222 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.705196 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-dshm\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.705443 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.705428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tls-certs\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.712090 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.712065 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jtl\" (UniqueName: \"kubernetes.io/projected/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-kube-api-access-r5jtl\") pod \"auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:10.732892 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:10.732870 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:11.011761 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:11.011731 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5"] Sep 17 17:55:11.016256 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:55:11.016221 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8830932_ebf5_4d6c_b5e8_2b0002443d2b.slice/crio-aea1ad541aa7cedbf4a2391d1fb64373330c27abde8d55831b574ab682659f8d WatchSource:0}: Error finding container aea1ad541aa7cedbf4a2391d1fb64373330c27abde8d55831b574ab682659f8d: Status 404 returned error can't find the container with id aea1ad541aa7cedbf4a2391d1fb64373330c27abde8d55831b574ab682659f8d Sep 17 17:55:11.907181 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:11.907138 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" event={"ID":"d8830932-ebf5-4d6c-b5e8-2b0002443d2b","Type":"ContainerStarted","Data":"5cd5aa186343ac0b941994c4e7c9ca4791781bd4a1bdeefd55541b9465ceab8c"} Sep 17 17:55:11.907697 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:11.907192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" event={"ID":"d8830932-ebf5-4d6c-b5e8-2b0002443d2b","Type":"ContainerStarted","Data":"aea1ad541aa7cedbf4a2391d1fb64373330c27abde8d55831b574ab682659f8d"} Sep 17 17:55:11.925302 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:11.925254 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" podStartSLOduration=1.9252398739999999 podStartE2EDuration="1.925239874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:11.923873973 +0000 UTC m=+1448.486038832" watchObservedRunningTime="2026-09-17 17:55:11.925239874 +0000 UTC m=+1448.487404734" Sep 17 17:55:20.733116 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:20.733084 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:20.733507 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:20.733140 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:20.746011 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:20.745984 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:55:20.967159 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:55:20.967126 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:57:02.582725 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.582691 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp"] Sep 17 17:57:02.586039 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.586023 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.592604 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.592580 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ab64d6-serving-cert\"" Sep 17 17:57:02.611308 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.611274 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rhq\" (UniqueName: \"kubernetes.io/projected/bb2b118a-5579-42fa-a9c5-38a2489cacad-kube-api-access-b8rhq\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.611472 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.611327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-maas-api-tls\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.611472 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.611414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2b118a-5579-42fa-a9c5-38a2489cacad-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.611554 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.611485 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-metrics-certs\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.619364 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.619334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp"] Sep 17 17:57:02.712362 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.712328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-metrics-certs\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.712544 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.712423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rhq\" (UniqueName: \"kubernetes.io/projected/bb2b118a-5579-42fa-a9c5-38a2489cacad-kube-api-access-b8rhq\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.712544 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.712453 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-maas-api-tls\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.712544 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.712480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2b118a-5579-42fa-a9c5-38a2489cacad-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.713245 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.713213 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2b118a-5579-42fa-a9c5-38a2489cacad-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.715014 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.714991 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-metrics-certs\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.715201 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.715182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-maas-api-tls\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.736300 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.736270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rhq\" (UniqueName: \"kubernetes.io/projected/bb2b118a-5579-42fa-a9c5-38a2489cacad-kube-api-access-b8rhq\") pod \"maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:02.897164 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:02.897073 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:03.210878 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:57:03.210842 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2b118a_5579_42fa_a9c5_38a2489cacad.slice/crio-420c6f6a05408e333ad55258013ef6ae140ddcade922b18ca2337f7616bd5d30 WatchSource:0}: Error finding container 420c6f6a05408e333ad55258013ef6ae140ddcade922b18ca2337f7616bd5d30: Status 404 returned error can't find the container with id 420c6f6a05408e333ad55258013ef6ae140ddcade922b18ca2337f7616bd5d30 Sep 17 17:57:03.211002 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:03.210975 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp"] Sep 17 17:57:03.503741 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:03.503649 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" event={"ID":"bb2b118a-5579-42fa-a9c5-38a2489cacad","Type":"ContainerStarted","Data":"420c6f6a05408e333ad55258013ef6ae140ddcade922b18ca2337f7616bd5d30"} Sep 17 17:57:04.511470 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:04.511400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" event={"ID":"bb2b118a-5579-42fa-a9c5-38a2489cacad","Type":"ContainerStarted","Data":"1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8"} Sep 17 17:57:04.512009 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:04.511508 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:04.547649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:04.547594 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" podStartSLOduration=2.029812642 podStartE2EDuration="2.547576997s" totalImagesPullingTime="517.764355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:04.545100149 +0000 UTC m=+1561.107265024" watchObservedRunningTime="2026-09-17 17:57:04.547576997 +0000 UTC m=+1561.109741857" Sep 17 17:57:10.522813 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:10.522780 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:57:13.541994 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:13.541941 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d"] Sep 17 17:57:13.555504 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:13.555473 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7"] Sep 17 17:57:13.594029 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:13.593996 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6"] Sep 17 17:57:17.586125 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:17.586086 2566 generic.go:356] "Generic (PLEG): container finished" podID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerID="a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e" exitCode=137 Sep 17 17:57:17.586538 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:17.586160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" event={"ID":"5daa7083-d1ec-4069-a7d6-9a3fec0574e6","Type":"ContainerDied","Data":"a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e"} Sep 17 17:57:17.586538 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:17.586533 2566 scope.go:117] "RemoveContainer" containerID="a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e" Sep 17 17:57:18.593546 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:18.593509 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" event={"ID":"5daa7083-d1ec-4069-a7d6-9a3fec0574e6","Type":"ContainerStarted","Data":"7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986"} Sep 17 17:57:18.593995 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:18.593739 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:57:28.562767 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.562722 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" podUID="fa80eefc-6844-47e2-ada0-9997376371c9" containerName="main" containerID="cri-o://db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7" gracePeriod=45 Sep 17 17:57:28.574646 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.574610 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" podUID="3ebae666-aa3e-4d74-8f29-a94e997c5b3c" containerName="main" containerID="cri-o://bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89" gracePeriod=45 Sep 17 17:57:28.613510 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.613473 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" podUID="38e13467-a5e2-4e01-ad08-b0b837795dcf" containerName="main" containerID="cri-o://8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892" gracePeriod=45 Sep 17 17:57:28.877437 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.877409 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:57:28.912714 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.912683 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:57:28.917211 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.917184 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:57:28.974166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974134 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vq6k\" (UniqueName: \"kubernetes.io/projected/fa80eefc-6844-47e2-ada0-9997376371c9-kube-api-access-6vq6k\") pod \"fa80eefc-6844-47e2-ada0-9997376371c9\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " Sep 17 17:57:28.974384 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974201 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-model-cache\") pod \"fa80eefc-6844-47e2-ada0-9997376371c9\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " Sep 17 17:57:28.974384 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974254 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-tmp-dir\") pod \"fa80eefc-6844-47e2-ada0-9997376371c9\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " Sep 17 17:57:28.974384 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974375 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-dshm\") pod \"fa80eefc-6844-47e2-ada0-9997376371c9\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " Sep 17 17:57:28.974573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974403 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa80eefc-6844-47e2-ada0-9997376371c9-tls-certs\") pod \"fa80eefc-6844-47e2-ada0-9997376371c9\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " Sep 17 17:57:28.974573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974433 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa80eefc-6844-47e2-ada0-9997376371c9" (UID: "fa80eefc-6844-47e2-ada0-9997376371c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:28.974573 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974502 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-home\") pod \"fa80eefc-6844-47e2-ada0-9997376371c9\" (UID: \"fa80eefc-6844-47e2-ada0-9997376371c9\") " Sep 17 17:57:28.974731 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974645 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa80eefc-6844-47e2-ada0-9997376371c9" (UID: "fa80eefc-6844-47e2-ada0-9997376371c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:28.974829 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974798 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-home" (OuterVolumeSpecName: "home") pod "fa80eefc-6844-47e2-ada0-9997376371c9" (UID: "fa80eefc-6844-47e2-ada0-9997376371c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:28.974971 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974931 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:28.975077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974977 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:28.975077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.974992 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:28.976723 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.976694 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-dshm" (OuterVolumeSpecName: "dshm") pod "fa80eefc-6844-47e2-ada0-9997376371c9" (UID: "fa80eefc-6844-47e2-ada0-9997376371c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:28.976828 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.976776 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa80eefc-6844-47e2-ada0-9997376371c9-kube-api-access-6vq6k" (OuterVolumeSpecName: "kube-api-access-6vq6k") pod "fa80eefc-6844-47e2-ada0-9997376371c9" (UID: "fa80eefc-6844-47e2-ada0-9997376371c9"). InnerVolumeSpecName "kube-api-access-6vq6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:57:28.976891 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:28.976876 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa80eefc-6844-47e2-ada0-9997376371c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa80eefc-6844-47e2-ada0-9997376371c9" (UID: "fa80eefc-6844-47e2-ada0-9997376371c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:57:29.076081 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-dshm\") pod \"38e13467-a5e2-4e01-ad08-b0b837795dcf\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " Sep 17 17:57:29.076278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076125 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tmp-dir\") pod \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " Sep 17 17:57:29.076278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076167 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7mlv\" (UniqueName: \"kubernetes.io/projected/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-kube-api-access-z7mlv\") pod \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " Sep 17 17:57:29.076278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076229 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-model-cache\") pod \"38e13467-a5e2-4e01-ad08-b0b837795dcf\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " Sep 17 17:57:29.076278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076252 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-tmp-dir\") pod \"38e13467-a5e2-4e01-ad08-b0b837795dcf\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076290 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz56m\" (UniqueName: \"kubernetes.io/projected/38e13467-a5e2-4e01-ad08-b0b837795dcf-kube-api-access-tz56m\") pod \"38e13467-a5e2-4e01-ad08-b0b837795dcf\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076311 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-dshm\") pod \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076343 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-home\") pod \"38e13467-a5e2-4e01-ad08-b0b837795dcf\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076390 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tls-certs\") pod \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076394 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ebae666-aa3e-4d74-8f29-a94e997c5b3c" (UID: "3ebae666-aa3e-4d74-8f29-a94e997c5b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076417 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-model-cache\") pod \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076461 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e13467-a5e2-4e01-ad08-b0b837795dcf-tls-certs\") pod \"38e13467-a5e2-4e01-ad08-b0b837795dcf\" (UID: \"38e13467-a5e2-4e01-ad08-b0b837795dcf\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076488 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-home\") pod \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\" (UID: \"3ebae666-aa3e-4d74-8f29-a94e997c5b3c\") " Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076526 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38e13467-a5e2-4e01-ad08-b0b837795dcf" (UID: "38e13467-a5e2-4e01-ad08-b0b837795dcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076540 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "38e13467-a5e2-4e01-ad08-b0b837795dcf" (UID: "38e13467-a5e2-4e01-ad08-b0b837795dcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076616 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-home" (OuterVolumeSpecName: "home") pod "38e13467-a5e2-4e01-ad08-b0b837795dcf" (UID: "38e13467-a5e2-4e01-ad08-b0b837795dcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076733 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ebae666-aa3e-4d74-8f29-a94e997c5b3c" (UID: "3ebae666-aa3e-4d74-8f29-a94e997c5b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.076874 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.076806 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-home" (OuterVolumeSpecName: "home") pod "3ebae666-aa3e-4d74-8f29-a94e997c5b3c" (UID: "3ebae666-aa3e-4d74-8f29-a94e997c5b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077171 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vq6k\" (UniqueName: \"kubernetes.io/projected/fa80eefc-6844-47e2-ada0-9997376371c9-kube-api-access-6vq6k\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077195 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077211 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077225 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077238 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa80eefc-6844-47e2-ada0-9997376371c9-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077251 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa80eefc-6844-47e2-ada0-9997376371c9-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077262 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077273 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.077769 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.077285 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.078807 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.078775 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-kube-api-access-z7mlv" (OuterVolumeSpecName: "kube-api-access-z7mlv") pod "3ebae666-aa3e-4d74-8f29-a94e997c5b3c" (UID: "3ebae666-aa3e-4d74-8f29-a94e997c5b3c"). InnerVolumeSpecName "kube-api-access-z7mlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:57:29.078923 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.078786 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-dshm" (OuterVolumeSpecName: "dshm") pod "38e13467-a5e2-4e01-ad08-b0b837795dcf" (UID: "38e13467-a5e2-4e01-ad08-b0b837795dcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.078923 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.078911 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ebae666-aa3e-4d74-8f29-a94e997c5b3c" (UID: "3ebae666-aa3e-4d74-8f29-a94e997c5b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:57:29.079073 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.079028 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-dshm" (OuterVolumeSpecName: "dshm") pod "3ebae666-aa3e-4d74-8f29-a94e997c5b3c" (UID: "3ebae666-aa3e-4d74-8f29-a94e997c5b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:57:29.079192 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.079173 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e13467-a5e2-4e01-ad08-b0b837795dcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38e13467-a5e2-4e01-ad08-b0b837795dcf" (UID: "38e13467-a5e2-4e01-ad08-b0b837795dcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:57:29.079306 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.079290 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e13467-a5e2-4e01-ad08-b0b837795dcf-kube-api-access-tz56m" (OuterVolumeSpecName: "kube-api-access-tz56m") pod "38e13467-a5e2-4e01-ad08-b0b837795dcf" (UID: "38e13467-a5e2-4e01-ad08-b0b837795dcf"). InnerVolumeSpecName "kube-api-access-tz56m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:57:29.178682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.178589 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7mlv\" (UniqueName: \"kubernetes.io/projected/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-kube-api-access-z7mlv\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.178682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.178638 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz56m\" (UniqueName: \"kubernetes.io/projected/38e13467-a5e2-4e01-ad08-b0b837795dcf-kube-api-access-tz56m\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.178682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.178649 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.178682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.178659 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ebae666-aa3e-4d74-8f29-a94e997c5b3c-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.178682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.178668 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e13467-a5e2-4e01-ad08-b0b837795dcf-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.178682 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.178675 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e13467-a5e2-4e01-ad08-b0b837795dcf-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.540566 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq"] Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541461 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa80eefc-6844-47e2-ada0-9997376371c9" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541484 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa80eefc-6844-47e2-ada0-9997376371c9" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541519 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebae666-aa3e-4d74-8f29-a94e997c5b3c" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541528 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebae666-aa3e-4d74-8f29-a94e997c5b3c" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541547 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e13467-a5e2-4e01-ad08-b0b837795dcf" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541555 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e13467-a5e2-4e01-ad08-b0b837795dcf" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541731 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebae666-aa3e-4d74-8f29-a94e997c5b3c" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541750 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e13467-a5e2-4e01-ad08-b0b837795dcf" containerName="main" Sep 17 17:57:29.543543 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.541771 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa80eefc-6844-47e2-ada0-9997376371c9" containerName="main" Sep 17 17:57:29.546900 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.546871 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.549506 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.549485 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-08f4f7-serving-cert\"" Sep 17 17:57:29.554981 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.554922 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq"] Sep 17 17:57:29.601167 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.601137 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 17:57:29.654788 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.654754 2566 generic.go:356] "Generic (PLEG): container finished" podID="fa80eefc-6844-47e2-ada0-9997376371c9" containerID="db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7" exitCode=0 Sep 17 17:57:29.654948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.654792 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" event={"ID":"fa80eefc-6844-47e2-ada0-9997376371c9","Type":"ContainerDied","Data":"db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7"} Sep 17 17:57:29.654948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.654811 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" Sep 17 17:57:29.654948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.654844 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d" event={"ID":"fa80eefc-6844-47e2-ada0-9997376371c9","Type":"ContainerDied","Data":"fd4b2b4fb3053dba0d7fe64dc896863db71776fb709f11334aa291d1a62ba0d8"} Sep 17 17:57:29.654948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.654869 2566 scope.go:117] "RemoveContainer" containerID="db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7" Sep 17 17:57:29.657242 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.656718 2566 generic.go:356] "Generic (PLEG): container finished" podID="3ebae666-aa3e-4d74-8f29-a94e997c5b3c" containerID="bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89" exitCode=0 Sep 17 17:57:29.657242 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.656784 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" Sep 17 17:57:29.657242 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.656862 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" event={"ID":"3ebae666-aa3e-4d74-8f29-a94e997c5b3c","Type":"ContainerDied","Data":"bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89"} Sep 17 17:57:29.657242 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.656916 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7" event={"ID":"3ebae666-aa3e-4d74-8f29-a94e997c5b3c","Type":"ContainerDied","Data":"dd828780821b1e8e29bce47aff16a63eb9435f809f633cb63267cbccae7eff85"} Sep 17 17:57:29.658895 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.658806 2566 generic.go:356] "Generic (PLEG): container finished" podID="38e13467-a5e2-4e01-ad08-b0b837795dcf" containerID="8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892" exitCode=0 Sep 17 17:57:29.658895 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.658847 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" event={"ID":"38e13467-a5e2-4e01-ad08-b0b837795dcf","Type":"ContainerDied","Data":"8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892"} Sep 17 17:57:29.658895 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.658875 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" event={"ID":"38e13467-a5e2-4e01-ad08-b0b837795dcf","Type":"ContainerDied","Data":"8883f2aa37ca12fb58c4b887c92548cea48b08660d76c73d635b0f3bde901c84"} Sep 17 17:57:29.658895 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.658880 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6" Sep 17 17:57:29.667532 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.667466 2566 scope.go:117] "RemoveContainer" containerID="db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7" Sep 17 17:57:29.667811 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:57:29.667784 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7\": container with ID starting with db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7 not found: ID does not exist" containerID="db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7" Sep 17 17:57:29.667919 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.667823 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7"} err="failed to get container status \"db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7\": rpc error: code = NotFound desc = could not find container \"db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7\": container with ID starting with db8497df7c788ee94bc8fcc7507f6affd4d9bce5d8d436483d1c939240d674c7 not found: ID does not exist" Sep 17 17:57:29.667919 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.667846 2566 scope.go:117] "RemoveContainer" containerID="bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89" Sep 17 17:57:29.678688 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.678668 2566 scope.go:117] "RemoveContainer" containerID="bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89" Sep 17 17:57:29.678988 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:57:29.678965 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89\": container with ID starting with bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89 not found: ID does not exist" containerID="bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89" Sep 17 17:57:29.679050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.678999 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89"} err="failed to get container status \"bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89\": rpc error: code = NotFound desc = could not find container \"bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89\": container with ID starting with bb537e6da71f65d75f10f14b06bcf07c7c62a482da350afa05c980902f396e89 not found: ID does not exist" Sep 17 17:57:29.679050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.679025 2566 scope.go:117] "RemoveContainer" containerID="8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892" Sep 17 17:57:29.683440 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.683408 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-metrics-certs\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.683542 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.683446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-maas-api-tls\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.683542 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.683518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-openshift-service-ca\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.683668 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.683638 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np5mx\" (UniqueName: \"kubernetes.io/projected/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-kube-api-access-np5mx\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.688138 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.688121 2566 scope.go:117] "RemoveContainer" containerID="8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892" Sep 17 17:57:29.688460 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:57:29.688434 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892\": container with ID starting with 8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892 not found: ID does not exist" containerID="8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892" Sep 17 17:57:29.688569 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.688495 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892"} err="failed to get container status \"8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892\": rpc error: code = NotFound desc = could not find container \"8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892\": container with ID starting with 8fc0e80971ebcce3589888d0191a0e4fbe1c7d5115a1be73ffe17868da2bb892 not found: ID does not exist" Sep 17 17:57:29.691901 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.691880 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d"] Sep 17 17:57:29.697060 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.697038 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-2-w0-db7813-kserve-7db46f8fcc-lpg9d"] Sep 17 17:57:29.714849 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.714826 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7"] Sep 17 17:57:29.718825 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.718799 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/distinct-w0-db7813-kserve-85ddf7878b-nd2l7"] Sep 17 17:57:29.739439 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.739415 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6"] Sep 17 17:57:29.749118 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.749098 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db7813/premium-w0-db7813-kserve-5498ff8899-pg2k6"] Sep 17 17:57:29.784333 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.784308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-metrics-certs\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.784462 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.784337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-maas-api-tls\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.784462 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.784406 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-openshift-service-ca\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.784561 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.784506 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np5mx\" (UniqueName: \"kubernetes.io/projected/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-kube-api-access-np5mx\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.785079 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.785054 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-openshift-service-ca\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.786721 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.786702 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-maas-api-tls\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.786814 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.786769 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-metrics-certs\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.798025 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.797970 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np5mx\" (UniqueName: \"kubernetes.io/projected/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-kube-api-access-np5mx\") pod \"maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:29.859028 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:29.858998 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:30.146414 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:57:30.146382 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfa0cff_74bc_44c1_aeb9_ad97f1a4c56b.slice/crio-fb398ca9e4bdcea40aaefaad939b04536e2f4db37606984de287a2dcb26e1182 WatchSource:0}: Error finding container fb398ca9e4bdcea40aaefaad939b04536e2f4db37606984de287a2dcb26e1182: Status 404 returned error can't find the container with id fb398ca9e4bdcea40aaefaad939b04536e2f4db37606984de287a2dcb26e1182 Sep 17 17:57:30.147794 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.147776 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:57:30.152410 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.152385 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e13467-a5e2-4e01-ad08-b0b837795dcf" path="/var/lib/kubelet/pods/38e13467-a5e2-4e01-ad08-b0b837795dcf/volumes" Sep 17 17:57:30.152808 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.152795 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebae666-aa3e-4d74-8f29-a94e997c5b3c" path="/var/lib/kubelet/pods/3ebae666-aa3e-4d74-8f29-a94e997c5b3c/volumes" Sep 17 17:57:30.153291 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.153272 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa80eefc-6844-47e2-ada0-9997376371c9" path="/var/lib/kubelet/pods/fa80eefc-6844-47e2-ada0-9997376371c9/volumes" Sep 17 17:57:30.153542 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.153530 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq"] Sep 17 17:57:30.629744 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.629709 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv"] Sep 17 17:57:30.632914 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.632897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.635879 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.635858 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-08f4f7\"" Sep 17 17:57:30.635995 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.635858 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-08f4f7-dockercfg-cdzjc\"" Sep 17 17:57:30.651837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.651813 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv"] Sep 17 17:57:30.667561 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.667527 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" event={"ID":"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b","Type":"ContainerStarted","Data":"fb398ca9e4bdcea40aaefaad939b04536e2f4db37606984de287a2dcb26e1182"} Sep 17 17:57:30.793525 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.793491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.793700 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.793548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnt8r\" (UniqueName: \"kubernetes.io/projected/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-kube-api-access-xnt8r\") pod \"payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.894867 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.894791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.894867 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.894843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnt8r\" (UniqueName: \"kubernetes.io/projected/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-kube-api-access-xnt8r\") pod \"payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.895618 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.895599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.904370 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.904344 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnt8r\" (UniqueName: \"kubernetes.io/projected/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-kube-api-access-xnt8r\") pod \"payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:30.943963 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:30.943925 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:31.232904 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.232875 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv"] Sep 17 17:57:31.234177 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:57:31.234151 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9293366_21bf_4d7f_ab07_18dbcc7e74b8.slice/crio-1879f5b8af4ed51188b3902fe13a63c0fd855574515aa06fd0aa62acffa427ee WatchSource:0}: Error finding container 1879f5b8af4ed51188b3902fe13a63c0fd855574515aa06fd0aa62acffa427ee: Status 404 returned error can't find the container with id 1879f5b8af4ed51188b3902fe13a63c0fd855574515aa06fd0aa62acffa427ee Sep 17 17:57:31.678613 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.678578 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" event={"ID":"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b","Type":"ContainerStarted","Data":"972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d"} Sep 17 17:57:31.679174 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.678646 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:31.680449 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.680424 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" event={"ID":"b9293366-21bf-4d7f-ab07-18dbcc7e74b8","Type":"ContainerStarted","Data":"854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b"} Sep 17 17:57:31.680577 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.680470 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:31.680577 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.680486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" event={"ID":"b9293366-21bf-4d7f-ab07-18dbcc7e74b8","Type":"ContainerStarted","Data":"1879f5b8af4ed51188b3902fe13a63c0fd855574515aa06fd0aa62acffa427ee"} Sep 17 17:57:31.723373 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.723328 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" podStartSLOduration=2.235824585 podStartE2EDuration="2.723312351s" totalImagesPullingTime="487.487766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:31.721278286 +0000 UTC m=+1588.283443145" watchObservedRunningTime="2026-09-17 17:57:31.723312351 +0000 UTC m=+1588.285477211" Sep 17 17:57:31.745423 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:31.745379 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" podStartSLOduration=1.745364263 podStartE2EDuration="1.745364263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:31.74348809 +0000 UTC m=+1588.305652950" watchObservedRunningTime="2026-09-17 17:57:31.745364263 +0000 UTC m=+1588.307529314" Sep 17 17:57:37.690863 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:37.690831 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:57:42.688361 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:42.688321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:57:59.736006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:59.735916 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp"] Sep 17 17:57:59.736479 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:59.736134 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" podUID="bb2b118a-5579-42fa-a9c5-38a2489cacad" containerName="maas-api" containerID="cri-o://1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8" gracePeriod=30 Sep 17 17:57:59.998502 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:57:59.998414 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:58:00.064040 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.064003 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-maas-api-tls\") pod \"bb2b118a-5579-42fa-a9c5-38a2489cacad\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " Sep 17 17:58:00.064233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.064126 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8rhq\" (UniqueName: \"kubernetes.io/projected/bb2b118a-5579-42fa-a9c5-38a2489cacad-kube-api-access-b8rhq\") pod \"bb2b118a-5579-42fa-a9c5-38a2489cacad\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " Sep 17 17:58:00.064233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.064167 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2b118a-5579-42fa-a9c5-38a2489cacad-openshift-service-ca\") pod \"bb2b118a-5579-42fa-a9c5-38a2489cacad\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " Sep 17 17:58:00.064233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.064223 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-metrics-certs\") pod \"bb2b118a-5579-42fa-a9c5-38a2489cacad\" (UID: \"bb2b118a-5579-42fa-a9c5-38a2489cacad\") " Sep 17 17:58:00.064562 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.064533 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2b118a-5579-42fa-a9c5-38a2489cacad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb2b118a-5579-42fa-a9c5-38a2489cacad" (UID: "bb2b118a-5579-42fa-a9c5-38a2489cacad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:58:00.066459 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.066421 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb2b118a-5579-42fa-a9c5-38a2489cacad" (UID: "bb2b118a-5579-42fa-a9c5-38a2489cacad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:58:00.066736 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.066709 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2b118a-5579-42fa-a9c5-38a2489cacad-kube-api-access-b8rhq" (OuterVolumeSpecName: "kube-api-access-b8rhq") pod "bb2b118a-5579-42fa-a9c5-38a2489cacad" (UID: "bb2b118a-5579-42fa-a9c5-38a2489cacad"). InnerVolumeSpecName "kube-api-access-b8rhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:58:00.066884 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.066860 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb2b118a-5579-42fa-a9c5-38a2489cacad" (UID: "bb2b118a-5579-42fa-a9c5-38a2489cacad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:58:00.165487 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.165459 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:00.165487 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.165487 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rhq\" (UniqueName: \"kubernetes.io/projected/bb2b118a-5579-42fa-a9c5-38a2489cacad-kube-api-access-b8rhq\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:00.165706 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.165497 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2b118a-5579-42fa-a9c5-38a2489cacad-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:00.165706 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.165505 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb2b118a-5579-42fa-a9c5-38a2489cacad-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:00.842673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.842638 2566 generic.go:356] "Generic (PLEG): container finished" podID="bb2b118a-5579-42fa-a9c5-38a2489cacad" containerID="1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8" exitCode=0 Sep 17 17:58:00.842673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.842675 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" event={"ID":"bb2b118a-5579-42fa-a9c5-38a2489cacad","Type":"ContainerDied","Data":"1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8"} Sep 17 17:58:00.843185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.842689 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" Sep 17 17:58:00.843185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.842716 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp" event={"ID":"bb2b118a-5579-42fa-a9c5-38a2489cacad","Type":"ContainerDied","Data":"420c6f6a05408e333ad55258013ef6ae140ddcade922b18ca2337f7616bd5d30"} Sep 17 17:58:00.843185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.842739 2566 scope.go:117] "RemoveContainer" containerID="1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8" Sep 17 17:58:00.851743 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.851724 2566 scope.go:117] "RemoveContainer" containerID="1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8" Sep 17 17:58:00.852030 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:58:00.852009 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8\": container with ID starting with 1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8 not found: ID does not exist" containerID="1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8" Sep 17 17:58:00.852087 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.852039 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8"} err="failed to get container status \"1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8\": rpc error: code = NotFound desc = could not find container \"1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8\": container with ID starting with 1971aba464bd39a14d3dc8f0eebe773eac8c586b90b0aec4c28e06c0469314a8 not found: ID does not exist" Sep 17 17:58:00.861078 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.861052 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp"] Sep 17 17:58:00.866564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:00.866539 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab64d6-56888cf6f5-l2skp"] Sep 17 17:58:02.150149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:02.150115 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2b118a-5579-42fa-a9c5-38a2489cacad" path="/var/lib/kubelet/pods/bb2b118a-5579-42fa-a9c5-38a2489cacad/volumes" Sep 17 17:58:25.448788 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.448754 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq"] Sep 17 17:58:25.449314 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.449018 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" podUID="adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" containerName="maas-api" containerID="cri-o://972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d" gracePeriod=30 Sep 17 17:58:25.530579 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.530300 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv"] Sep 17 17:58:25.532274 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.531053 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" podUID="b9293366-21bf-4d7f-ab07-18dbcc7e74b8" containerName="payload-pre-processing" containerID="cri-o://854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b" gracePeriod=30 Sep 17 17:58:25.809114 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.809088 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:58:25.813018 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.812999 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:58:25.925363 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925329 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnt8r\" (UniqueName: \"kubernetes.io/projected/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-kube-api-access-xnt8r\") pod \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " Sep 17 17:58:25.925513 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925397 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-openshift-service-ca\") pod \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " Sep 17 17:58:25.925513 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925413 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np5mx\" (UniqueName: \"kubernetes.io/projected/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-kube-api-access-np5mx\") pod \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " Sep 17 17:58:25.925513 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925457 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-metrics-certs\") pod \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " Sep 17 17:58:25.925513 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925491 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-plugins-config-volume\") pod \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\" (UID: \"b9293366-21bf-4d7f-ab07-18dbcc7e74b8\") " Sep 17 17:58:25.925513 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925510 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-maas-api-tls\") pod \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\" (UID: \"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b\") " Sep 17 17:58:25.925827 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925725 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" (UID: "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:58:25.925890 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925845 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:25.925979 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.925884 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9293366-21bf-4d7f-ab07-18dbcc7e74b8" (UID: "b9293366-21bf-4d7f-ab07-18dbcc7e74b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:58:25.927650 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.927615 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-kube-api-access-np5mx" (OuterVolumeSpecName: "kube-api-access-np5mx") pod "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" (UID: "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b"). InnerVolumeSpecName "kube-api-access-np5mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:58:25.927650 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.927636 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" (UID: "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:58:25.927801 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.927656 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-kube-api-access-xnt8r" (OuterVolumeSpecName: "kube-api-access-xnt8r") pod "b9293366-21bf-4d7f-ab07-18dbcc7e74b8" (UID: "b9293366-21bf-4d7f-ab07-18dbcc7e74b8"). InnerVolumeSpecName "kube-api-access-xnt8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:58:25.927801 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.927719 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" (UID: "adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:58:25.976736 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.976701 2566 generic.go:356] "Generic (PLEG): container finished" podID="b9293366-21bf-4d7f-ab07-18dbcc7e74b8" containerID="854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b" exitCode=0 Sep 17 17:58:25.976868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.976749 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" Sep 17 17:58:25.976868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.976795 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" event={"ID":"b9293366-21bf-4d7f-ab07-18dbcc7e74b8","Type":"ContainerDied","Data":"854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b"} Sep 17 17:58:25.976868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.976858 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv" event={"ID":"b9293366-21bf-4d7f-ab07-18dbcc7e74b8","Type":"ContainerDied","Data":"1879f5b8af4ed51188b3902fe13a63c0fd855574515aa06fd0aa62acffa427ee"} Sep 17 17:58:25.977033 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.976887 2566 scope.go:117] "RemoveContainer" containerID="854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b" Sep 17 17:58:25.978301 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.978271 2566 generic.go:356] "Generic (PLEG): container finished" podID="adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" containerID="972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d" exitCode=0 Sep 17 17:58:25.978398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.978303 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" event={"ID":"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b","Type":"ContainerDied","Data":"972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d"} Sep 17 17:58:25.978398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.978326 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" Sep 17 17:58:25.978398 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.978354 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq" event={"ID":"adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b","Type":"ContainerDied","Data":"fb398ca9e4bdcea40aaefaad939b04536e2f4db37606984de287a2dcb26e1182"} Sep 17 17:58:25.988526 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.988490 2566 scope.go:117] "RemoveContainer" containerID="854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b" Sep 17 17:58:25.988796 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:58:25.988778 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b\": container with ID starting with 854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b not found: ID does not exist" containerID="854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b" Sep 17 17:58:25.988868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.988802 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b"} err="failed to get container status \"854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b\": rpc error: code = NotFound desc = could not find container \"854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b\": container with ID starting with 854d4ec6ad5b1a730364e770493fb18706eb69dbd4bd182b2369733931a4133b not found: ID does not exist" Sep 17 17:58:25.988868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.988819 2566 scope.go:117] "RemoveContainer" containerID="972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d" Sep 17 17:58:25.996841 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.996824 2566 scope.go:117] "RemoveContainer" containerID="972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d" Sep 17 17:58:25.997142 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:58:25.997122 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d\": container with ID starting with 972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d not found: ID does not exist" containerID="972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d" Sep 17 17:58:25.997204 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:25.997149 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d"} err="failed to get container status \"972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d\": rpc error: code = NotFound desc = could not find container \"972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d\": container with ID starting with 972c220752a716692452c37ab2788213aeb3225f50630ef7710e6d05cbb9266d not found: ID does not exist" Sep 17 17:58:26.007640 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.007617 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv"] Sep 17 17:58:26.009625 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.009606 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-08f4f7-64bf74cb66-gsrgv"] Sep 17 17:58:26.026687 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.026663 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np5mx\" (UniqueName: \"kubernetes.io/projected/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-kube-api-access-np5mx\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:26.026760 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.026688 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:26.026760 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.026703 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:26.026760 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.026716 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:26.026760 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.026742 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnt8r\" (UniqueName: \"kubernetes.io/projected/b9293366-21bf-4d7f-ab07-18dbcc7e74b8-kube-api-access-xnt8r\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:26.028571 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.028549 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq"] Sep 17 17:58:26.033080 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.033058 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08f4f7-f4c7db7b4-fd2xq"] Sep 17 17:58:26.151851 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.151817 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" path="/var/lib/kubelet/pods/adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b/volumes" Sep 17 17:58:26.152251 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:26.152235 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9293366-21bf-4d7f-ab07-18dbcc7e74b8" path="/var/lib/kubelet/pods/b9293366-21bf-4d7f-ab07-18dbcc7e74b8/volumes" Sep 17 17:58:45.068873 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.068835 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm"] Sep 17 17:58:45.069427 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.069132 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" podUID="022026e5-e89e-487f-ba40-f04307e1365c" containerName="payload-pre-processing" containerID="cri-o://33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5" gracePeriod=30 Sep 17 17:58:45.071845 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.071817 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp"] Sep 17 17:58:45.072115 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.072067 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" podUID="87ec3fbe-0ade-4297-ac79-a87bd7b589be" containerName="payload-processing" containerID="cri-o://33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2" gracePeriod=30 Sep 17 17:58:45.362071 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.362037 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:58:45.364983 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.364962 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:58:45.514989 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.514947 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf88w\" (UniqueName: \"kubernetes.io/projected/87ec3fbe-0ade-4297-ac79-a87bd7b589be-kube-api-access-jf88w\") pod \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " Sep 17 17:58:45.515153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.515069 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxjjj\" (UniqueName: \"kubernetes.io/projected/022026e5-e89e-487f-ba40-f04307e1365c-kube-api-access-dxjjj\") pod \"022026e5-e89e-487f-ba40-f04307e1365c\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " Sep 17 17:58:45.515153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.515091 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022026e5-e89e-487f-ba40-f04307e1365c-plugins-config-volume\") pod \"022026e5-e89e-487f-ba40-f04307e1365c\" (UID: \"022026e5-e89e-487f-ba40-f04307e1365c\") " Sep 17 17:58:45.515153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.515150 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ec3fbe-0ade-4297-ac79-a87bd7b589be-plugins-config-volume\") pod \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\" (UID: \"87ec3fbe-0ade-4297-ac79-a87bd7b589be\") " Sep 17 17:58:45.515522 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.515496 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022026e5-e89e-487f-ba40-f04307e1365c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "022026e5-e89e-487f-ba40-f04307e1365c" (UID: "022026e5-e89e-487f-ba40-f04307e1365c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:58:45.515634 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.515553 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ec3fbe-0ade-4297-ac79-a87bd7b589be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87ec3fbe-0ade-4297-ac79-a87bd7b589be" (UID: "87ec3fbe-0ade-4297-ac79-a87bd7b589be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:58:45.517149 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.517126 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022026e5-e89e-487f-ba40-f04307e1365c-kube-api-access-dxjjj" (OuterVolumeSpecName: "kube-api-access-dxjjj") pod "022026e5-e89e-487f-ba40-f04307e1365c" (UID: "022026e5-e89e-487f-ba40-f04307e1365c"). InnerVolumeSpecName "kube-api-access-dxjjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:58:45.517233 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.517154 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ec3fbe-0ade-4297-ac79-a87bd7b589be-kube-api-access-jf88w" (OuterVolumeSpecName: "kube-api-access-jf88w") pod "87ec3fbe-0ade-4297-ac79-a87bd7b589be" (UID: "87ec3fbe-0ade-4297-ac79-a87bd7b589be"). InnerVolumeSpecName "kube-api-access-jf88w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:58:45.616756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.616682 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjjj\" (UniqueName: \"kubernetes.io/projected/022026e5-e89e-487f-ba40-f04307e1365c-kube-api-access-dxjjj\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:45.616756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.616708 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022026e5-e89e-487f-ba40-f04307e1365c-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:45.616756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.616720 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ec3fbe-0ade-4297-ac79-a87bd7b589be-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:45.616756 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:45.616728 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf88w\" (UniqueName: \"kubernetes.io/projected/87ec3fbe-0ade-4297-ac79-a87bd7b589be-kube-api-access-jf88w\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:58:46.093354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.093319 2566 generic.go:356] "Generic (PLEG): container finished" podID="022026e5-e89e-487f-ba40-f04307e1365c" containerID="33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5" exitCode=0 Sep 17 17:58:46.093815 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.093367 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" Sep 17 17:58:46.093815 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.093399 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" event={"ID":"022026e5-e89e-487f-ba40-f04307e1365c","Type":"ContainerDied","Data":"33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5"} Sep 17 17:58:46.093815 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.093456 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm" event={"ID":"022026e5-e89e-487f-ba40-f04307e1365c","Type":"ContainerDied","Data":"a1c307a5755f8fa6fef87c5ac6333d3f6cdfcb468d085261b23d525048f02977"} Sep 17 17:58:46.093815 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.093474 2566 scope.go:117] "RemoveContainer" containerID="33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5" Sep 17 17:58:46.094817 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.094794 2566 generic.go:356] "Generic (PLEG): container finished" podID="87ec3fbe-0ade-4297-ac79-a87bd7b589be" containerID="33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2" exitCode=0 Sep 17 17:58:46.094946 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.094823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" event={"ID":"87ec3fbe-0ade-4297-ac79-a87bd7b589be","Type":"ContainerDied","Data":"33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2"} Sep 17 17:58:46.094946 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.094851 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" Sep 17 17:58:46.094946 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.094880 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp" event={"ID":"87ec3fbe-0ade-4297-ac79-a87bd7b589be","Type":"ContainerDied","Data":"12df58023a355c484e5e26fdf42e3fa059cb525aac78a4fd1ff79f9ca8ae9db1"} Sep 17 17:58:46.102429 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.102401 2566 scope.go:117] "RemoveContainer" containerID="33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5" Sep 17 17:58:46.102714 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:58:46.102693 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5\": container with ID starting with 33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5 not found: ID does not exist" containerID="33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5" Sep 17 17:58:46.102807 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.102721 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5"} err="failed to get container status \"33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5\": rpc error: code = NotFound desc = could not find container \"33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5\": container with ID starting with 33d916ba734439301cf5c2216fdc1ac6176c64b23c61df84aed92dc3317dd1d5 not found: ID does not exist" Sep 17 17:58:46.102807 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.102745 2566 scope.go:117] "RemoveContainer" containerID="33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2" Sep 17 17:58:46.116854 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.116838 2566 scope.go:117] "RemoveContainer" containerID="33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2" Sep 17 17:58:46.117151 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:58:46.117130 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2\": container with ID starting with 33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2 not found: ID does not exist" containerID="33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2" Sep 17 17:58:46.117226 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.117162 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2"} err="failed to get container status \"33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2\": rpc error: code = NotFound desc = could not find container \"33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2\": container with ID starting with 33537d5df38ab5701fc6d68f94e8e13dbce5fd10381308c41f8e8811646ab2e2 not found: ID does not exist" Sep 17 17:58:46.149183 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.149157 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm"] Sep 17 17:58:46.161603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.161583 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7e8755-749f7b8c74-59lnm"] Sep 17 17:58:46.188180 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.188153 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp"] Sep 17 17:58:46.192456 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:46.192435 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7e8755-6c65f6cccf-d7cpp"] Sep 17 17:58:48.150343 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:48.150311 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="022026e5-e89e-487f-ba40-f04307e1365c" path="/var/lib/kubelet/pods/022026e5-e89e-487f-ba40-f04307e1365c/volumes" Sep 17 17:58:48.150723 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:48.150705 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ec3fbe-0ade-4297-ac79-a87bd7b589be" path="/var/lib/kubelet/pods/87ec3fbe-0ade-4297-ac79-a87bd7b589be/volumes" Sep 17 17:58:57.158305 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:57.158268 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7"] Sep 17 17:58:57.175099 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:57.175068 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h"] Sep 17 17:58:57.185849 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:58:57.185821 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79"] Sep 17 17:59:04.880042 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:04.880007 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t"] Sep 17 17:59:04.880511 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:04.880258 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" podUID="c14c0eec-b53a-4df9-aac9-38a71f2f939c" containerName="payload-pre-processing" containerID="cri-o://f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee" gracePeriod=30 Sep 17 17:59:05.133500 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.133442 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:59:05.197206 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.197172 2566 generic.go:356] "Generic (PLEG): container finished" podID="c14c0eec-b53a-4df9-aac9-38a71f2f939c" containerID="f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee" exitCode=0 Sep 17 17:59:05.197206 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.197207 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" Sep 17 17:59:05.197435 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.197234 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" event={"ID":"c14c0eec-b53a-4df9-aac9-38a71f2f939c","Type":"ContainerDied","Data":"f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee"} Sep 17 17:59:05.197435 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.197284 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t" event={"ID":"c14c0eec-b53a-4df9-aac9-38a71f2f939c","Type":"ContainerDied","Data":"84163ae91dff8921d4c434f73d01cddcda00dd3f491ac1784bddd5f70d87b179"} Sep 17 17:59:05.197435 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.197317 2566 scope.go:117] "RemoveContainer" containerID="f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee" Sep 17 17:59:05.202719 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.202698 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6h97\" (UniqueName: \"kubernetes.io/projected/c14c0eec-b53a-4df9-aac9-38a71f2f939c-kube-api-access-z6h97\") pod \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " Sep 17 17:59:05.202840 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.202804 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14c0eec-b53a-4df9-aac9-38a71f2f939c-plugins-config-volume\") pod \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\" (UID: \"c14c0eec-b53a-4df9-aac9-38a71f2f939c\") " Sep 17 17:59:05.203223 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.203199 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14c0eec-b53a-4df9-aac9-38a71f2f939c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c14c0eec-b53a-4df9-aac9-38a71f2f939c" (UID: "c14c0eec-b53a-4df9-aac9-38a71f2f939c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:59:05.205140 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.205100 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14c0eec-b53a-4df9-aac9-38a71f2f939c-kube-api-access-z6h97" (OuterVolumeSpecName: "kube-api-access-z6h97") pod "c14c0eec-b53a-4df9-aac9-38a71f2f939c" (UID: "c14c0eec-b53a-4df9-aac9-38a71f2f939c"). InnerVolumeSpecName "kube-api-access-z6h97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:05.207117 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.206921 2566 scope.go:117] "RemoveContainer" containerID="f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee" Sep 17 17:59:05.207299 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:05.207281 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee\": container with ID starting with f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee not found: ID does not exist" containerID="f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee" Sep 17 17:59:05.207352 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.207309 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee"} err="failed to get container status \"f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee\": rpc error: code = NotFound desc = could not find container \"f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee\": container with ID starting with f1ed38b48ef752ec3322aea1b236cfa2797c8dd6ac9cfa4f8d87075e52159bee not found: ID does not exist" Sep 17 17:59:05.304118 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.304082 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6h97\" (UniqueName: \"kubernetes.io/projected/c14c0eec-b53a-4df9-aac9-38a71f2f939c-kube-api-access-z6h97\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:05.304118 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.304115 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14c0eec-b53a-4df9-aac9-38a71f2f939c-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:05.527691 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.527655 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t"] Sep 17 17:59:05.537924 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:05.537900 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef41f8-5d7f79c9cd-7dn8t"] Sep 17 17:59:06.150334 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.150298 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14c0eec-b53a-4df9-aac9-38a71f2f939c" path="/var/lib/kubelet/pods/c14c0eec-b53a-4df9-aac9-38a71f2f939c/volumes" Sep 17 17:59:06.420984 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.420880 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6"] Sep 17 17:59:06.421160 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.421136 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" podUID="567dc582-bcdd-482b-9dc2-3e747952d4d2" containerName="maas-api" containerID="cri-o://4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762" gracePeriod=30 Sep 17 17:59:06.876603 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.876577 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:59:06.918076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.918044 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls\") pod \"567dc582-bcdd-482b-9dc2-3e747952d4d2\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " Sep 17 17:59:06.918076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.918080 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p649q\" (UniqueName: \"kubernetes.io/projected/567dc582-bcdd-482b-9dc2-3e747952d4d2-kube-api-access-p649q\") pod \"567dc582-bcdd-482b-9dc2-3e747952d4d2\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " Sep 17 17:59:06.918276 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.918143 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-metrics-certs\") pod \"567dc582-bcdd-482b-9dc2-3e747952d4d2\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " Sep 17 17:59:06.918276 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.918175 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567dc582-bcdd-482b-9dc2-3e747952d4d2-openshift-service-ca\") pod \"567dc582-bcdd-482b-9dc2-3e747952d4d2\" (UID: \"567dc582-bcdd-482b-9dc2-3e747952d4d2\") " Sep 17 17:59:06.918579 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.918555 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567dc582-bcdd-482b-9dc2-3e747952d4d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "567dc582-bcdd-482b-9dc2-3e747952d4d2" (UID: "567dc582-bcdd-482b-9dc2-3e747952d4d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:59:06.920811 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.920674 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "567dc582-bcdd-482b-9dc2-3e747952d4d2" (UID: "567dc582-bcdd-482b-9dc2-3e747952d4d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:06.921485 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.921456 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "567dc582-bcdd-482b-9dc2-3e747952d4d2" (UID: "567dc582-bcdd-482b-9dc2-3e747952d4d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:06.921485 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:06.921489 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567dc582-bcdd-482b-9dc2-3e747952d4d2-kube-api-access-p649q" (OuterVolumeSpecName: "kube-api-access-p649q") pod "567dc582-bcdd-482b-9dc2-3e747952d4d2" (UID: "567dc582-bcdd-482b-9dc2-3e747952d4d2"). InnerVolumeSpecName "kube-api-access-p649q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:07.019093 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.019004 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:07.019093 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.019033 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p649q\" (UniqueName: \"kubernetes.io/projected/567dc582-bcdd-482b-9dc2-3e747952d4d2-kube-api-access-p649q\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:07.019093 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.019046 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567dc582-bcdd-482b-9dc2-3e747952d4d2-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:07.019093 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.019062 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567dc582-bcdd-482b-9dc2-3e747952d4d2-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:07.210166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.210132 2566 generic.go:356] "Generic (PLEG): container finished" podID="567dc582-bcdd-482b-9dc2-3e747952d4d2" containerID="4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762" exitCode=0 Sep 17 17:59:07.210166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.210175 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" Sep 17 17:59:07.210675 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.210214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" event={"ID":"567dc582-bcdd-482b-9dc2-3e747952d4d2","Type":"ContainerDied","Data":"4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762"} Sep 17 17:59:07.210675 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.210268 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6" event={"ID":"567dc582-bcdd-482b-9dc2-3e747952d4d2","Type":"ContainerDied","Data":"60dc1a2c5d1a9e18d41ff95a4cf3d1d6b79bbc4955a1a4a829243f9723593925"} Sep 17 17:59:07.210675 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.210285 2566 scope.go:117] "RemoveContainer" containerID="4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762" Sep 17 17:59:07.219868 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.219849 2566 scope.go:117] "RemoveContainer" containerID="4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762" Sep 17 17:59:07.220187 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:07.220167 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762\": container with ID starting with 4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762 not found: ID does not exist" containerID="4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762" Sep 17 17:59:07.220265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.220198 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762"} err="failed to get container status \"4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762\": rpc error: code = NotFound desc = could not find container \"4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762\": container with ID starting with 4d34db4b3e91b5668fad3503f80e9c39ce76fef74aa26aca48c949a494fbb762 not found: ID does not exist" Sep 17 17:59:07.239924 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.239900 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6"] Sep 17 17:59:07.253950 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:07.253924 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8aeea1-7974df7c87-mtst6"] Sep 17 17:59:08.150128 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:08.150095 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567dc582-bcdd-482b-9dc2-3e747952d4d2" path="/var/lib/kubelet/pods/567dc582-bcdd-482b-9dc2-3e747952d4d2/volumes" Sep 17 17:59:12.178964 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.178922 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" podUID="52e7f93e-fe69-4a77-87d2-42a5154bd2ac" containerName="main" containerID="cri-o://f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f" gracePeriod=45 Sep 17 17:59:12.200530 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.200469 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" podUID="337d2a80-65f5-42a5-a841-44127c42b351" containerName="main" containerID="cri-o://92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e" gracePeriod=45 Sep 17 17:59:12.205039 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.205005 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" podUID="159ff45e-9ede-4787-b935-17910da974f7" containerName="main" containerID="cri-o://70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba" gracePeriod=45 Sep 17 17:59:12.472982 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.468547 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:59:12.499211 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.499163 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:59:12.503122 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.503096 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:59:12.575662 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575630 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tmp-dir\") pod \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " Sep 17 17:59:12.575824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575677 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mb2z\" (UniqueName: \"kubernetes.io/projected/337d2a80-65f5-42a5-a841-44127c42b351-kube-api-access-6mb2z\") pod \"337d2a80-65f5-42a5-a841-44127c42b351\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " Sep 17 17:59:12.575824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575725 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-home\") pod \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " Sep 17 17:59:12.575824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575756 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-tmp-dir\") pod \"159ff45e-9ede-4787-b935-17910da974f7\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " Sep 17 17:59:12.575824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575781 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-model-cache\") pod \"159ff45e-9ede-4787-b935-17910da974f7\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " Sep 17 17:59:12.575824 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575814 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-model-cache\") pod \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575854 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-tmp-dir\") pod \"337d2a80-65f5-42a5-a841-44127c42b351\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575886 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tls-certs\") pod \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575917 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-dshm\") pod \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575942 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337d2a80-65f5-42a5-a841-44127c42b351-tls-certs\") pod \"337d2a80-65f5-42a5-a841-44127c42b351\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575991 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-dshm\") pod \"159ff45e-9ede-4787-b935-17910da974f7\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.575976 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52e7f93e-fe69-4a77-87d2-42a5154bd2ac" (UID: "52e7f93e-fe69-4a77-87d2-42a5154bd2ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576013 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-home\") pod \"337d2a80-65f5-42a5-a841-44127c42b351\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576106 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ff45e-9ede-4787-b935-17910da974f7-tls-certs\") pod \"159ff45e-9ede-4787-b935-17910da974f7\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " Sep 17 17:59:12.576142 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576132 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-dshm\") pod \"337d2a80-65f5-42a5-a841-44127c42b351\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " Sep 17 17:59:12.576630 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576170 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbg6m\" (UniqueName: \"kubernetes.io/projected/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-kube-api-access-tbg6m\") pod \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\" (UID: \"52e7f93e-fe69-4a77-87d2-42a5154bd2ac\") " Sep 17 17:59:12.576630 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576214 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-model-cache\") pod \"337d2a80-65f5-42a5-a841-44127c42b351\" (UID: \"337d2a80-65f5-42a5-a841-44127c42b351\") " Sep 17 17:59:12.576630 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576239 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5btj\" (UniqueName: \"kubernetes.io/projected/159ff45e-9ede-4787-b935-17910da974f7-kube-api-access-m5btj\") pod \"159ff45e-9ede-4787-b935-17910da974f7\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " Sep 17 17:59:12.576630 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576265 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-home\") pod \"159ff45e-9ede-4787-b935-17910da974f7\" (UID: \"159ff45e-9ede-4787-b935-17910da974f7\") " Sep 17 17:59:12.576630 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576609 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.576892 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576689 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-home" (OuterVolumeSpecName: "home") pod "52e7f93e-fe69-4a77-87d2-42a5154bd2ac" (UID: "52e7f93e-fe69-4a77-87d2-42a5154bd2ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.576892 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576814 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-home" (OuterVolumeSpecName: "home") pod "159ff45e-9ede-4787-b935-17910da974f7" (UID: "159ff45e-9ede-4787-b935-17910da974f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.577030 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.576896 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "52e7f93e-fe69-4a77-87d2-42a5154bd2ac" (UID: "52e7f93e-fe69-4a77-87d2-42a5154bd2ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.577030 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.577020 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-home" (OuterVolumeSpecName: "home") pod "337d2a80-65f5-42a5-a841-44127c42b351" (UID: "337d2a80-65f5-42a5-a841-44127c42b351"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.577170 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.577144 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "159ff45e-9ede-4787-b935-17910da974f7" (UID: "159ff45e-9ede-4787-b935-17910da974f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.577354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.577333 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "337d2a80-65f5-42a5-a841-44127c42b351" (UID: "337d2a80-65f5-42a5-a841-44127c42b351"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.577467 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.577443 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "159ff45e-9ede-4787-b935-17910da974f7" (UID: "159ff45e-9ede-4787-b935-17910da974f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.577676 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.577651 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-model-cache" (OuterVolumeSpecName: "model-cache") pod "337d2a80-65f5-42a5-a841-44127c42b351" (UID: "337d2a80-65f5-42a5-a841-44127c42b351"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.579470 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.579441 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159ff45e-9ede-4787-b935-17910da974f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "159ff45e-9ede-4787-b935-17910da974f7" (UID: "159ff45e-9ede-4787-b935-17910da974f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:12.579970 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.579919 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-dshm" (OuterVolumeSpecName: "dshm") pod "52e7f93e-fe69-4a77-87d2-42a5154bd2ac" (UID: "52e7f93e-fe69-4a77-87d2-42a5154bd2ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.580399 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.580322 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-dshm" (OuterVolumeSpecName: "dshm") pod "159ff45e-9ede-4787-b935-17910da974f7" (UID: "159ff45e-9ede-4787-b935-17910da974f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.580399 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.580373 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337d2a80-65f5-42a5-a841-44127c42b351-kube-api-access-6mb2z" (OuterVolumeSpecName: "kube-api-access-6mb2z") pod "337d2a80-65f5-42a5-a841-44127c42b351" (UID: "337d2a80-65f5-42a5-a841-44127c42b351"). InnerVolumeSpecName "kube-api-access-6mb2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:12.580567 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.580428 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159ff45e-9ede-4787-b935-17910da974f7-kube-api-access-m5btj" (OuterVolumeSpecName: "kube-api-access-m5btj") pod "159ff45e-9ede-4787-b935-17910da974f7" (UID: "159ff45e-9ede-4787-b935-17910da974f7"). InnerVolumeSpecName "kube-api-access-m5btj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:12.581069 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.580838 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-dshm" (OuterVolumeSpecName: "dshm") pod "337d2a80-65f5-42a5-a841-44127c42b351" (UID: "337d2a80-65f5-42a5-a841-44127c42b351"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:12.581327 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.581304 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-kube-api-access-tbg6m" (OuterVolumeSpecName: "kube-api-access-tbg6m") pod "52e7f93e-fe69-4a77-87d2-42a5154bd2ac" (UID: "52e7f93e-fe69-4a77-87d2-42a5154bd2ac"). InnerVolumeSpecName "kube-api-access-tbg6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:12.581504 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.581481 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52e7f93e-fe69-4a77-87d2-42a5154bd2ac" (UID: "52e7f93e-fe69-4a77-87d2-42a5154bd2ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:12.581933 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.581918 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337d2a80-65f5-42a5-a841-44127c42b351-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "337d2a80-65f5-42a5-a841-44127c42b351" (UID: "337d2a80-65f5-42a5-a841-44127c42b351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:12.677935 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677901 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.677935 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677931 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.677935 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677941 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677949 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337d2a80-65f5-42a5-a841-44127c42b351-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677973 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677984 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.677994 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ff45e-9ede-4787-b935-17910da974f7-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678006 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678018 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbg6m\" (UniqueName: \"kubernetes.io/projected/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-kube-api-access-tbg6m\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678028 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337d2a80-65f5-42a5-a841-44127c42b351-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678037 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5btj\" (UniqueName: \"kubernetes.io/projected/159ff45e-9ede-4787-b935-17910da974f7-kube-api-access-m5btj\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678047 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678055 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mb2z\" (UniqueName: \"kubernetes.io/projected/337d2a80-65f5-42a5-a841-44127c42b351-kube-api-access-6mb2z\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678062 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678071 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678078 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ff45e-9ede-4787-b935-17910da974f7-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:12.678166 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:12.678087 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e7f93e-fe69-4a77-87d2-42a5154bd2ac-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:13.240461 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.240427 2566 generic.go:356] "Generic (PLEG): container finished" podID="52e7f93e-fe69-4a77-87d2-42a5154bd2ac" containerID="f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f" exitCode=0 Sep 17 17:59:13.240837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.240473 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" Sep 17 17:59:13.240837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.240515 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" event={"ID":"52e7f93e-fe69-4a77-87d2-42a5154bd2ac","Type":"ContainerDied","Data":"f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f"} Sep 17 17:59:13.240837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.240560 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7" event={"ID":"52e7f93e-fe69-4a77-87d2-42a5154bd2ac","Type":"ContainerDied","Data":"d7d18c72b6271c6662840ccef30b5bc2c740eb99fc34c122d737dfaa12ff4dd7"} Sep 17 17:59:13.240837 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.240586 2566 scope.go:117] "RemoveContainer" containerID="f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f" Sep 17 17:59:13.241946 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.241923 2566 generic.go:356] "Generic (PLEG): container finished" podID="159ff45e-9ede-4787-b935-17910da974f7" containerID="70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba" exitCode=0 Sep 17 17:59:13.242047 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.241966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" event={"ID":"159ff45e-9ede-4787-b935-17910da974f7","Type":"ContainerDied","Data":"70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba"} Sep 17 17:59:13.242047 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.241987 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" Sep 17 17:59:13.242047 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.242004 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79" event={"ID":"159ff45e-9ede-4787-b935-17910da974f7","Type":"ContainerDied","Data":"90a207a85e082408bee7e9325be1c0de1a217ea2dee88a9fc6cc7bdb374a4e70"} Sep 17 17:59:13.243500 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.243476 2566 generic.go:356] "Generic (PLEG): container finished" podID="337d2a80-65f5-42a5-a841-44127c42b351" containerID="92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e" exitCode=0 Sep 17 17:59:13.243629 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.243509 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" Sep 17 17:59:13.243629 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.243508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" event={"ID":"337d2a80-65f5-42a5-a841-44127c42b351","Type":"ContainerDied","Data":"92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e"} Sep 17 17:59:13.243629 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.243619 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h" event={"ID":"337d2a80-65f5-42a5-a841-44127c42b351","Type":"ContainerDied","Data":"1b1a2dfbb01dce8aff7b7e4e7d09a8ddb71ee4896f6b4a260d67e83939ae8d89"} Sep 17 17:59:13.250673 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.250652 2566 scope.go:117] "RemoveContainer" containerID="f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f" Sep 17 17:59:13.250949 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:13.250931 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f\": container with ID starting with f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f not found: ID does not exist" containerID="f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f" Sep 17 17:59:13.251043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.250972 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f"} err="failed to get container status \"f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f\": rpc error: code = NotFound desc = could not find container \"f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f\": container with ID starting with f01d659e7e1251accaaa9c115e4936e9563da7251c039f8424de0c5640acda1f not found: ID does not exist" Sep 17 17:59:13.251043 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.250993 2566 scope.go:117] "RemoveContainer" containerID="70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba" Sep 17 17:59:13.258779 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.258765 2566 scope.go:117] "RemoveContainer" containerID="70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba" Sep 17 17:59:13.259032 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:13.259016 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba\": container with ID starting with 70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba not found: ID does not exist" containerID="70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba" Sep 17 17:59:13.259077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.259039 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba"} err="failed to get container status \"70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba\": rpc error: code = NotFound desc = could not find container \"70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba\": container with ID starting with 70a9df11b78597c9da3e2a003da16122da5b077a117ae9ce72fc861f6a961dba not found: ID does not exist" Sep 17 17:59:13.259077 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.259054 2566 scope.go:117] "RemoveContainer" containerID="92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e" Sep 17 17:59:13.265171 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.265148 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7"] Sep 17 17:59:13.266564 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.266534 2566 scope.go:117] "RemoveContainer" containerID="92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e" Sep 17 17:59:13.266861 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:13.266843 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e\": container with ID starting with 92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e not found: ID does not exist" containerID="92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e" Sep 17 17:59:13.266928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.266867 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e"} err="failed to get container status \"92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e\": rpc error: code = NotFound desc = could not find container \"92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e\": container with ID starting with 92af09a3cdff58c211ec772659b2b9728b78b9b018c5004567d9c237ff6bf98e not found: ID does not exist" Sep 17 17:59:13.271668 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.271647 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/distinct-w4-8aeea1-kserve-955f7bc76-d5rl7"] Sep 17 17:59:13.283626 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.283591 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h"] Sep 17 17:59:13.288596 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.288561 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/premium-w4-8aeea1-kserve-68f78f8789-cms6h"] Sep 17 17:59:13.303642 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.303620 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79"] Sep 17 17:59:13.308375 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:13.308352 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8aeea1/simulator-w4-8aeea1-kserve-fb769c9b8-f6n79"] Sep 17 17:59:14.149197 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:14.149166 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="159ff45e-9ede-4787-b935-17910da974f7" path="/var/lib/kubelet/pods/159ff45e-9ede-4787-b935-17910da974f7/volumes" Sep 17 17:59:14.149661 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:14.149643 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337d2a80-65f5-42a5-a841-44127c42b351" path="/var/lib/kubelet/pods/337d2a80-65f5-42a5-a841-44127c42b351/volumes" Sep 17 17:59:14.150076 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:14.150062 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e7f93e-fe69-4a77-87d2-42a5154bd2ac" path="/var/lib/kubelet/pods/52e7f93e-fe69-4a77-87d2-42a5154bd2ac/volumes" Sep 17 17:59:24.854661 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.854624 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-4bcd29"] Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.854974 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159ff45e-9ede-4787-b935-17910da974f7" containerName="main" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.854987 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="159ff45e-9ede-4787-b935-17910da974f7" containerName="main" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.854995 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9293366-21bf-4d7f-ab07-18dbcc7e74b8" containerName="payload-pre-processing" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855001 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9293366-21bf-4d7f-ab07-18dbcc7e74b8" containerName="payload-pre-processing" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855010 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337d2a80-65f5-42a5-a841-44127c42b351" containerName="main" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855015 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="337d2a80-65f5-42a5-a841-44127c42b351" containerName="main" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855020 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14c0eec-b53a-4df9-aac9-38a71f2f939c" containerName="payload-pre-processing" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855027 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14c0eec-b53a-4df9-aac9-38a71f2f939c" containerName="payload-pre-processing" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855035 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e7f93e-fe69-4a77-87d2-42a5154bd2ac" containerName="main" Sep 17 17:59:24.855050 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855040 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e7f93e-fe69-4a77-87d2-42a5154bd2ac" containerName="main" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855057 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2b118a-5579-42fa-a9c5-38a2489cacad" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855062 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2b118a-5579-42fa-a9c5-38a2489cacad" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855070 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855075 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855082 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ec3fbe-0ade-4297-ac79-a87bd7b589be" containerName="payload-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855087 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ec3fbe-0ade-4297-ac79-a87bd7b589be" containerName="payload-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855095 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022026e5-e89e-487f-ba40-f04307e1365c" containerName="payload-pre-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855100 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="022026e5-e89e-487f-ba40-f04307e1365c" containerName="payload-pre-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855108 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567dc582-bcdd-482b-9dc2-3e747952d4d2" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855114 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="567dc582-bcdd-482b-9dc2-3e747952d4d2" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855169 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="337d2a80-65f5-42a5-a841-44127c42b351" containerName="main" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855174 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9293366-21bf-4d7f-ab07-18dbcc7e74b8" containerName="payload-pre-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855180 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14c0eec-b53a-4df9-aac9-38a71f2f939c" containerName="payload-pre-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855188 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="022026e5-e89e-487f-ba40-f04307e1365c" containerName="payload-pre-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855193 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="159ff45e-9ede-4787-b935-17910da974f7" containerName="main" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855198 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2b118a-5579-42fa-a9c5-38a2489cacad" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855204 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfa0cff-74bc-44c1-aeb9-ad97f1a4c56b" containerName="maas-api" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855210 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ec3fbe-0ade-4297-ac79-a87bd7b589be" containerName="payload-processing" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855216 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e7f93e-fe69-4a77-87d2-42a5154bd2ac" containerName="main" Sep 17 17:59:24.855360 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.855221 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="567dc582-bcdd-482b-9dc2-3e747952d4d2" containerName="maas-api" Sep 17 17:59:24.859690 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.859674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:24.864544 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.864523 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z7xqm\"" Sep 17 17:59:24.871991 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.871948 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-4bcd29"] Sep 17 17:59:24.991810 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:24.991773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75tv2\" (UniqueName: \"kubernetes.io/projected/8aa5f745-eb36-4ad9-9b96-b002aaf291a7-kube-api-access-75tv2\") pod \"test-curl-7603-4bcd29\" (UID: \"8aa5f745-eb36-4ad9-9b96-b002aaf291a7\") " pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:25.092650 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:25.092614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75tv2\" (UniqueName: \"kubernetes.io/projected/8aa5f745-eb36-4ad9-9b96-b002aaf291a7-kube-api-access-75tv2\") pod \"test-curl-7603-4bcd29\" (UID: \"8aa5f745-eb36-4ad9-9b96-b002aaf291a7\") " pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:25.111499 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:25.111423 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75tv2\" (UniqueName: \"kubernetes.io/projected/8aa5f745-eb36-4ad9-9b96-b002aaf291a7-kube-api-access-75tv2\") pod \"test-curl-7603-4bcd29\" (UID: \"8aa5f745-eb36-4ad9-9b96-b002aaf291a7\") " pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:25.169368 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:25.169340 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:25.447104 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:25.447012 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-4bcd29"] Sep 17 17:59:25.450147 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:59:25.450115 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aa5f745_eb36_4ad9_9b96_b002aaf291a7.slice/crio-b9e5d45fc8c4739e754e807796e9c925b2dffc9baf894b9bdd3d8a707f2c8851 WatchSource:0}: Error finding container b9e5d45fc8c4739e754e807796e9c925b2dffc9baf894b9bdd3d8a707f2c8851: Status 404 returned error can't find the container with id b9e5d45fc8c4739e754e807796e9c925b2dffc9baf894b9bdd3d8a707f2c8851 Sep 17 17:59:26.320473 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:26.320443 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-4bcd29" event={"ID":"8aa5f745-eb36-4ad9-9b96-b002aaf291a7","Type":"ContainerStarted","Data":"f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0"} Sep 17 17:59:26.320473 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:26.320481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-4bcd29" event={"ID":"8aa5f745-eb36-4ad9-9b96-b002aaf291a7","Type":"ContainerStarted","Data":"b9e5d45fc8c4739e754e807796e9c925b2dffc9baf894b9bdd3d8a707f2c8851"} Sep 17 17:59:26.340496 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:26.340418 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7603-4bcd29" podStartSLOduration=2.340403739 podStartE2EDuration="2.340403739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:26.338733247 +0000 UTC m=+1702.900898108" watchObservedRunningTime="2026-09-17 17:59:26.340403739 +0000 UTC m=+1702.902568602" Sep 17 17:59:26.987550 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:26.987513 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-4bcd29"] Sep 17 17:59:26.991948 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:26.991922 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-4bcd29"] Sep 17 17:59:27.325649 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:27.325617 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7603-4bcd29" container="test-curl-7603-4bcd29" Sep 17 17:59:28.331115 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:28.331079 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7603-4bcd29" podUID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" containerName="test-curl-7603-4bcd29" containerID="cri-o://f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0" gracePeriod=2 Sep 17 17:59:28.639088 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:28.638996 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s"] Sep 17 17:59:30.266454 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:30.266417 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5"] Sep 17 17:59:30.487656 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:30.487633 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:30.644951 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:30.644919 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75tv2\" (UniqueName: \"kubernetes.io/projected/8aa5f745-eb36-4ad9-9b96-b002aaf291a7-kube-api-access-75tv2\") pod \"8aa5f745-eb36-4ad9-9b96-b002aaf291a7\" (UID: \"8aa5f745-eb36-4ad9-9b96-b002aaf291a7\") " Sep 17 17:59:30.647052 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:30.647029 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa5f745-eb36-4ad9-9b96-b002aaf291a7-kube-api-access-75tv2" (OuterVolumeSpecName: "kube-api-access-75tv2") pod "8aa5f745-eb36-4ad9-9b96-b002aaf291a7" (UID: "8aa5f745-eb36-4ad9-9b96-b002aaf291a7"). InnerVolumeSpecName "kube-api-access-75tv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:30.746357 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:30.746319 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75tv2\" (UniqueName: \"kubernetes.io/projected/8aa5f745-eb36-4ad9-9b96-b002aaf291a7-kube-api-access-75tv2\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:31.347624 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:31.347595 2566 generic.go:356] "Generic (PLEG): container finished" podID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" containerID="f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0" exitCode=137 Sep 17 17:59:31.347997 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:31.347637 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:31.347997 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:31.347662 2566 scope.go:117] "RemoveContainer" containerID="f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0" Sep 17 17:59:31.356732 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:31.356719 2566 scope.go:117] "RemoveContainer" containerID="f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0" Sep 17 17:59:31.356970 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:31.356938 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0\": container with ID starting with f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0 not found: ID does not exist" containerID="f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0" Sep 17 17:59:31.357019 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:31.356976 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0"} err="failed to get container status \"f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0\": rpc error: code = NotFound desc = could not find container \"f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0\": container with ID starting with f736159944e5398d75881249811e3192df00e609f8000d231cb7292a9ad140d0 not found: ID does not exist" Sep 17 17:59:31.358373 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:31.358356 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-4bcd29\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" pod="opendatahub/test-curl-7603-4bcd29" Sep 17 17:59:32.149500 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:32.149459 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" path="/var/lib/kubelet/pods/8aa5f745-eb36-4ad9-9b96-b002aaf291a7/volumes" Sep 17 17:59:43.661832 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.661772 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" podUID="66fb4f1d-067f-490c-8eda-734c9e7bcb37" containerName="main" containerID="cri-o://586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48" gracePeriod=45 Sep 17 17:59:43.914262 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.914209 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:59:43.956208 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956176 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29rmr\" (UniqueName: \"kubernetes.io/projected/66fb4f1d-067f-490c-8eda-734c9e7bcb37-kube-api-access-29rmr\") pod \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " Sep 17 17:59:43.956208 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956207 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-dshm\") pod \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " Sep 17 17:59:43.956424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956234 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tmp-dir\") pod \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " Sep 17 17:59:43.956424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956293 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-model-cache\") pod \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " Sep 17 17:59:43.956424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956328 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-home\") pod \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " Sep 17 17:59:43.956424 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956369 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tls-certs\") pod \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\" (UID: \"66fb4f1d-067f-490c-8eda-734c9e7bcb37\") " Sep 17 17:59:43.956649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956500 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66fb4f1d-067f-490c-8eda-734c9e7bcb37" (UID: "66fb4f1d-067f-490c-8eda-734c9e7bcb37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:43.956649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956541 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-model-cache" (OuterVolumeSpecName: "model-cache") pod "66fb4f1d-067f-490c-8eda-734c9e7bcb37" (UID: "66fb4f1d-067f-490c-8eda-734c9e7bcb37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:43.956649 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956618 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-home" (OuterVolumeSpecName: "home") pod "66fb4f1d-067f-490c-8eda-734c9e7bcb37" (UID: "66fb4f1d-067f-490c-8eda-734c9e7bcb37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:43.956816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956709 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:43.956816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956727 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:43.956816 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.956741 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:43.958464 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.958437 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-dshm" (OuterVolumeSpecName: "dshm") pod "66fb4f1d-067f-490c-8eda-734c9e7bcb37" (UID: "66fb4f1d-067f-490c-8eda-734c9e7bcb37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:43.958570 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.958446 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fb4f1d-067f-490c-8eda-734c9e7bcb37-kube-api-access-29rmr" (OuterVolumeSpecName: "kube-api-access-29rmr") pod "66fb4f1d-067f-490c-8eda-734c9e7bcb37" (UID: "66fb4f1d-067f-490c-8eda-734c9e7bcb37"). InnerVolumeSpecName "kube-api-access-29rmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:43.958570 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:43.958512 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66fb4f1d-067f-490c-8eda-734c9e7bcb37" (UID: "66fb4f1d-067f-490c-8eda-734c9e7bcb37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:44.058038 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.058015 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fb4f1d-067f-490c-8eda-734c9e7bcb37-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:44.058153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.058039 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rmr\" (UniqueName: \"kubernetes.io/projected/66fb4f1d-067f-490c-8eda-734c9e7bcb37-kube-api-access-29rmr\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:44.058153 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.058053 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fb4f1d-067f-490c-8eda-734c9e7bcb37-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:44.416408 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.416377 2566 generic.go:356] "Generic (PLEG): container finished" podID="66fb4f1d-067f-490c-8eda-734c9e7bcb37" containerID="586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48" exitCode=0 Sep 17 17:59:44.416582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.416433 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" Sep 17 17:59:44.416582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.416438 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" event={"ID":"66fb4f1d-067f-490c-8eda-734c9e7bcb37","Type":"ContainerDied","Data":"586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48"} Sep 17 17:59:44.416582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.416483 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s" event={"ID":"66fb4f1d-067f-490c-8eda-734c9e7bcb37","Type":"ContainerDied","Data":"7c9ecb1d8e30045a1527bfcd9656a0926d4af0b46bec1b6da4ec406b28944f1b"} Sep 17 17:59:44.416582 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.416501 2566 scope.go:117] "RemoveContainer" containerID="586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48" Sep 17 17:59:44.424532 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.424513 2566 scope.go:117] "RemoveContainer" containerID="586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48" Sep 17 17:59:44.424801 ip-10-0-142-117 kubenswrapper[2566]: E0917 17:59:44.424778 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48\": container with ID starting with 586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48 not found: ID does not exist" containerID="586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48" Sep 17 17:59:44.424879 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.424810 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48"} err="failed to get container status \"586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48\": rpc error: code = NotFound desc = could not find container \"586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48\": container with ID starting with 586429282e71cc36f88f22e43363c753d441ee262a5e7306bd626f287ee2be48 not found: ID does not exist" Sep 17 17:59:44.435159 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.435137 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s"] Sep 17 17:59:44.438278 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:44.438259 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/auth-test-model-abb363-kserve-6c46ffbfbc-tpn7s"] Sep 17 17:59:45.285309 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.285269 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" podUID="d8830932-ebf5-4d6c-b5e8-2b0002443d2b" containerName="main" containerID="cri-o://5cd5aa186343ac0b941994c4e7c9ca4791781bd4a1bdeefd55541b9465ceab8c" gracePeriod=45 Sep 17 17:59:45.423238 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.423212 2566 generic.go:356] "Generic (PLEG): container finished" podID="d8830932-ebf5-4d6c-b5e8-2b0002443d2b" containerID="5cd5aa186343ac0b941994c4e7c9ca4791781bd4a1bdeefd55541b9465ceab8c" exitCode=0 Sep 17 17:59:45.423372 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.423294 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" event={"ID":"d8830932-ebf5-4d6c-b5e8-2b0002443d2b","Type":"ContainerDied","Data":"5cd5aa186343ac0b941994c4e7c9ca4791781bd4a1bdeefd55541b9465ceab8c"} Sep 17 17:59:45.532746 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.532725 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:59:45.570727 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.570705 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5jtl\" (UniqueName: \"kubernetes.io/projected/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-kube-api-access-r5jtl\") pod \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " Sep 17 17:59:45.570928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.570736 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tmp-dir\") pod \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " Sep 17 17:59:45.570928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.570757 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tls-certs\") pod \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " Sep 17 17:59:45.570928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.570784 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-dshm\") pod \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " Sep 17 17:59:45.570928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.570827 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-model-cache\") pod \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " Sep 17 17:59:45.570928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.570856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-home\") pod \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\" (UID: \"d8830932-ebf5-4d6c-b5e8-2b0002443d2b\") " Sep 17 17:59:45.571248 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.571041 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8830932-ebf5-4d6c-b5e8-2b0002443d2b" (UID: "d8830932-ebf5-4d6c-b5e8-2b0002443d2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:45.571248 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.571141 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:45.571354 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.571255 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-home" (OuterVolumeSpecName: "home") pod "d8830932-ebf5-4d6c-b5e8-2b0002443d2b" (UID: "d8830932-ebf5-4d6c-b5e8-2b0002443d2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:45.571434 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.571396 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8830932-ebf5-4d6c-b5e8-2b0002443d2b" (UID: "d8830932-ebf5-4d6c-b5e8-2b0002443d2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:45.573081 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.573053 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8830932-ebf5-4d6c-b5e8-2b0002443d2b" (UID: "d8830932-ebf5-4d6c-b5e8-2b0002443d2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:59:45.573196 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.573073 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-kube-api-access-r5jtl" (OuterVolumeSpecName: "kube-api-access-r5jtl") pod "d8830932-ebf5-4d6c-b5e8-2b0002443d2b" (UID: "d8830932-ebf5-4d6c-b5e8-2b0002443d2b"). InnerVolumeSpecName "kube-api-access-r5jtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:59:45.573327 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.573310 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-dshm" (OuterVolumeSpecName: "dshm") pod "d8830932-ebf5-4d6c-b5e8-2b0002443d2b" (UID: "d8830932-ebf5-4d6c-b5e8-2b0002443d2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:59:45.671591 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.671560 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jtl\" (UniqueName: \"kubernetes.io/projected/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-kube-api-access-r5jtl\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:45.671591 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.671586 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:45.671591 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.671598 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:45.671768 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.671606 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:45.671768 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:45.671614 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8830932-ebf5-4d6c-b5e8-2b0002443d2b-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 17:59:46.150011 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:46.149981 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66fb4f1d-067f-490c-8eda-734c9e7bcb37" path="/var/lib/kubelet/pods/66fb4f1d-067f-490c-8eda-734c9e7bcb37/volumes" Sep 17 17:59:46.429928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:46.429847 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" Sep 17 17:59:46.429928 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:46.429895 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5" event={"ID":"d8830932-ebf5-4d6c-b5e8-2b0002443d2b","Type":"ContainerDied","Data":"aea1ad541aa7cedbf4a2391d1fb64373330c27abde8d55831b574ab682659f8d"} Sep 17 17:59:46.430459 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:46.429946 2566 scope.go:117] "RemoveContainer" containerID="5cd5aa186343ac0b941994c4e7c9ca4791781bd4a1bdeefd55541b9465ceab8c" Sep 17 17:59:46.452747 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:46.452726 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5"] Sep 17 17:59:46.455021 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:46.455003 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/auth-test-model-7b7c8c-kserve-8656b84f55-hwfp5"] Sep 17 17:59:48.150041 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:48.150001 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8830932-ebf5-4d6c-b5e8-2b0002443d2b" path="/var/lib/kubelet/pods/d8830932-ebf5-4d6c-b5e8-2b0002443d2b/volumes" Sep 17 17:59:53.251753 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.251721 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr"] Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252084 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8830932-ebf5-4d6c-b5e8-2b0002443d2b" containerName="main" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252096 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8830932-ebf5-4d6c-b5e8-2b0002443d2b" containerName="main" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252110 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" containerName="test-curl-7603-4bcd29" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252116 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" containerName="test-curl-7603-4bcd29" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252126 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66fb4f1d-067f-490c-8eda-734c9e7bcb37" containerName="main" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252131 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66fb4f1d-067f-490c-8eda-734c9e7bcb37" containerName="main" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252184 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="66fb4f1d-067f-490c-8eda-734c9e7bcb37" containerName="main" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252194 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8830932-ebf5-4d6c-b5e8-2b0002443d2b" containerName="main" Sep 17 17:59:53.252265 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.252200 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5f745-eb36-4ad9-9b96-b002aaf291a7" containerName="test-curl-7603-4bcd29" Sep 17 17:59:53.256415 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.256395 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.261006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.260928 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"default-dockercfg-k9lpm\"" Sep 17 17:59:53.261006 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.260939 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"openshift-service-ca.crt\"" Sep 17 17:59:53.261185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.261009 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"kube-root-ca.crt\"" Sep 17 17:59:53.261185 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.261024 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-abb363\"/\"sub-test-model-abb363-kserve-self-signed-certs\"" Sep 17 17:59:53.267903 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.267879 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr"] Sep 17 17:59:53.331480 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.331451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tmp-dir\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.331641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.331494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-home\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.331641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.331536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlg7q\" (UniqueName: \"kubernetes.io/projected/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-kube-api-access-wlg7q\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.331641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.331575 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-model-cache\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.331641 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.331596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tls-certs\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.331791 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.331670 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-dshm\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.432784 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.432748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlg7q\" (UniqueName: \"kubernetes.io/projected/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-kube-api-access-wlg7q\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.432992 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.432794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-model-cache\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.432992 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.432814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tls-certs\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.432992 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.432848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-dshm\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.432992 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.432875 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tmp-dir\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.432992 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.432891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-home\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.433315 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.433292 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-model-cache\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.433381 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.433331 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-home\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.433381 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.433349 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tmp-dir\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.435138 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.435120 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-dshm\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.435325 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.435309 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tls-certs\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.447083 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.447063 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlg7q\" (UniqueName: \"kubernetes.io/projected/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-kube-api-access-wlg7q\") pod \"sub-test-model-abb363-kserve-75db759d49-zgsrr\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.565309 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.565280 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 17:59:53.843067 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:53.843044 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr"] Sep 17 17:59:53.844924 ip-10-0-142-117 kubenswrapper[2566]: W0917 17:59:53.844900 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f3b5a3b_7b98_45a8_934b_cb3b6c5ed669.slice/crio-5affe9ebc849ad088846be07502c998a33bfa477e7263ffd94bcdc614f2b250f WatchSource:0}: Error finding container 5affe9ebc849ad088846be07502c998a33bfa477e7263ffd94bcdc614f2b250f: Status 404 returned error can't find the container with id 5affe9ebc849ad088846be07502c998a33bfa477e7263ffd94bcdc614f2b250f Sep 17 17:59:54.471432 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:54.471388 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" event={"ID":"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669","Type":"ContainerStarted","Data":"a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793"} Sep 17 17:59:54.471812 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:54.471447 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" event={"ID":"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669","Type":"ContainerStarted","Data":"5affe9ebc849ad088846be07502c998a33bfa477e7263ffd94bcdc614f2b250f"} Sep 17 17:59:54.492654 ip-10-0-142-117 kubenswrapper[2566]: I0917 17:59:54.492607 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" podStartSLOduration=1.492591725 podStartE2EDuration="1.492591725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:54.491810211 +0000 UTC m=+1731.053975071" watchObservedRunningTime="2026-09-17 17:59:54.492591725 +0000 UTC m=+1731.054756585" Sep 17 18:00:03.565718 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:03.565678 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 18:00:03.566349 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:03.565733 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 18:00:03.578315 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:03.578279 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 18:00:04.531475 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:04.531386 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 18:00:15.342658 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.342626 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml"] Sep 17 18:00:15.346199 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.346180 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.348926 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.348906 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"kube-root-ca.crt\"" Sep 17 18:00:15.349631 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.349612 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"default-dockercfg-n6kdw\"" Sep 17 18:00:15.349723 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.349709 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"sub-test-model-7b7c8c-kserve-self-signed-certs\"" Sep 17 18:00:15.349806 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.349792 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"openshift-service-ca.crt\"" Sep 17 18:00:15.354846 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.354825 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml"] Sep 17 18:00:15.433454 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.433424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-model-cache\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.433616 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.433480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2v2\" (UniqueName: \"kubernetes.io/projected/bf0cd405-8bd1-4599-9078-516b75d3c77b-kube-api-access-5r2v2\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.433616 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.433505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-home\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.433616 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.433561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-tmp-dir\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.433616 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.433600 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-dshm\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.433764 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.433627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0cd405-8bd1-4599-9078-516b75d3c77b-tls-certs\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.534732 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.534695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-model-cache\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.534927 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.534760 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2v2\" (UniqueName: \"kubernetes.io/projected/bf0cd405-8bd1-4599-9078-516b75d3c77b-kube-api-access-5r2v2\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.534927 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.534795 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-home\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.534927 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.534840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-tmp-dir\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.534927 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.534912 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-dshm\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.535177 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.534945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0cd405-8bd1-4599-9078-516b75d3c77b-tls-certs\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.535238 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.535179 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-model-cache\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.535238 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.535198 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-tmp-dir\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.535335 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.535263 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-home\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.537163 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.537143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-dshm\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.537475 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.537454 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0cd405-8bd1-4599-9078-516b75d3c77b-tls-certs\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.543414 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.543391 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2v2\" (UniqueName: \"kubernetes.io/projected/bf0cd405-8bd1-4599-9078-516b75d3c77b-kube-api-access-5r2v2\") pod \"sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.655138 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.655045 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:15.933950 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:15.933801 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml"] Sep 17 18:00:15.936785 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:00:15.936746 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf0cd405_8bd1_4599_9078_516b75d3c77b.slice/crio-48982abd6a2ac6a21d61b7fae3c961adba18e9009885a06b972411f052b983b6 WatchSource:0}: Error finding container 48982abd6a2ac6a21d61b7fae3c961adba18e9009885a06b972411f052b983b6: Status 404 returned error can't find the container with id 48982abd6a2ac6a21d61b7fae3c961adba18e9009885a06b972411f052b983b6 Sep 17 18:00:16.578669 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:16.578632 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" event={"ID":"bf0cd405-8bd1-4599-9078-516b75d3c77b","Type":"ContainerStarted","Data":"ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4"} Sep 17 18:00:16.579075 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:16.578678 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" event={"ID":"bf0cd405-8bd1-4599-9078-516b75d3c77b","Type":"ContainerStarted","Data":"48982abd6a2ac6a21d61b7fae3c961adba18e9009885a06b972411f052b983b6"} Sep 17 18:00:16.605016 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:16.604943 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" podStartSLOduration=1.604925192 podStartE2EDuration="1.604925192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:16.603830068 +0000 UTC m=+1753.165994927" watchObservedRunningTime="2026-09-17 18:00:16.604925192 +0000 UTC m=+1753.167090056" Sep 17 18:00:25.656233 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:25.656190 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:25.656733 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:25.656252 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:25.668844 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:25.668812 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:26.640052 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:26.640018 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:00:50.414413 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.414371 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-dfa2b3"] Sep 17 18:00:50.418119 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.418096 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:50.422412 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.422393 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z7xqm\"" Sep 17 18:00:50.429097 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.429076 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-dfa2b3"] Sep 17 18:00:50.529554 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.529514 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmbp\" (UniqueName: \"kubernetes.io/projected/d646c80a-96bc-4c59-806d-033f8bacbfed-kube-api-access-dxmbp\") pod \"test-curl-7603-dfa2b3\" (UID: \"d646c80a-96bc-4c59-806d-033f8bacbfed\") " pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:50.630554 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.630516 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmbp\" (UniqueName: \"kubernetes.io/projected/d646c80a-96bc-4c59-806d-033f8bacbfed-kube-api-access-dxmbp\") pod \"test-curl-7603-dfa2b3\" (UID: \"d646c80a-96bc-4c59-806d-033f8bacbfed\") " pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:50.640934 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.640909 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmbp\" (UniqueName: \"kubernetes.io/projected/d646c80a-96bc-4c59-806d-033f8bacbfed-kube-api-access-dxmbp\") pod \"test-curl-7603-dfa2b3\" (UID: \"d646c80a-96bc-4c59-806d-033f8bacbfed\") " pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:50.727450 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:50.727357 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:51.000628 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:51.000603 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-dfa2b3"] Sep 17 18:00:51.002924 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:00:51.002897 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd646c80a_96bc_4c59_806d_033f8bacbfed.slice/crio-3f6e5b3d86d8cd3aeb76e1114cc626b4db2e24ab596a6e5991a7d5e1c19c9a74 WatchSource:0}: Error finding container 3f6e5b3d86d8cd3aeb76e1114cc626b4db2e24ab596a6e5991a7d5e1c19c9a74: Status 404 returned error can't find the container with id 3f6e5b3d86d8cd3aeb76e1114cc626b4db2e24ab596a6e5991a7d5e1c19c9a74 Sep 17 18:00:51.750508 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:51.750474 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-dfa2b3" event={"ID":"d646c80a-96bc-4c59-806d-033f8bacbfed","Type":"ContainerStarted","Data":"648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3"} Sep 17 18:00:51.750508 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:51.750513 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-dfa2b3" event={"ID":"d646c80a-96bc-4c59-806d-033f8bacbfed","Type":"ContainerStarted","Data":"3f6e5b3d86d8cd3aeb76e1114cc626b4db2e24ab596a6e5991a7d5e1c19c9a74"} Sep 17 18:00:51.768207 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:51.768162 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7603-dfa2b3" podStartSLOduration=1.7681465269999999 podStartE2EDuration="1.768146527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:51.76722118 +0000 UTC m=+1788.329386041" watchObservedRunningTime="2026-09-17 18:00:51.768146527 +0000 UTC m=+1788.330311394" Sep 17 18:00:52.357305 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.357270 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-dfa2b3"] Sep 17 18:00:52.362002 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.361976 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-dfa2b3"] Sep 17 18:00:52.483714 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.483630 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-5266f4"] Sep 17 18:00:52.484021 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.484008 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" containerName="test-curl-7603-dfa2b3" Sep 17 18:00:52.484073 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.484022 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" containerName="test-curl-7603-dfa2b3" Sep 17 18:00:52.484115 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.484094 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" containerName="test-curl-7603-dfa2b3" Sep 17 18:00:52.487005 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.486985 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:52.494774 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.494745 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-5266f4"] Sep 17 18:00:52.545860 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.545835 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jf2\" (UniqueName: \"kubernetes.io/projected/5db02186-9c54-4578-bb93-07a14a8287f8-kube-api-access-c5jf2\") pod \"test-curl-7603-5266f4\" (UID: \"5db02186-9c54-4578-bb93-07a14a8287f8\") " pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:52.647200 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.647166 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jf2\" (UniqueName: \"kubernetes.io/projected/5db02186-9c54-4578-bb93-07a14a8287f8-kube-api-access-c5jf2\") pod \"test-curl-7603-5266f4\" (UID: \"5db02186-9c54-4578-bb93-07a14a8287f8\") " pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:52.655923 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.655892 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jf2\" (UniqueName: \"kubernetes.io/projected/5db02186-9c54-4578-bb93-07a14a8287f8-kube-api-access-c5jf2\") pod \"test-curl-7603-5266f4\" (UID: \"5db02186-9c54-4578-bb93-07a14a8287f8\") " pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:52.754982 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:52.754902 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7603-dfa2b3" container="test-curl-7603-dfa2b3" Sep 17 18:00:52.796909 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:52.796885 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:53.070280 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:53.070227 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-5266f4"] Sep 17 18:00:53.075162 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:00:53.075130 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5db02186_9c54_4578_bb93_07a14a8287f8.slice/crio-02288033e514a3b5dd015ba6cad70f651a94f707eb97bebdacca91ced6f301d5 WatchSource:0}: Error finding container 02288033e514a3b5dd015ba6cad70f651a94f707eb97bebdacca91ced6f301d5: Status 404 returned error can't find the container with id 02288033e514a3b5dd015ba6cad70f651a94f707eb97bebdacca91ced6f301d5 Sep 17 18:00:53.760595 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:53.760561 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-5266f4" event={"ID":"5db02186-9c54-4578-bb93-07a14a8287f8","Type":"ContainerStarted","Data":"61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a"} Sep 17 18:00:53.761077 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:53.760603 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-5266f4" event={"ID":"5db02186-9c54-4578-bb93-07a14a8287f8","Type":"ContainerStarted","Data":"02288033e514a3b5dd015ba6cad70f651a94f707eb97bebdacca91ced6f301d5"} Sep 17 18:00:53.761077 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:53.760656 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7603-dfa2b3" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" containerName="test-curl-7603-dfa2b3" containerID="cri-o://648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3" gracePeriod=2 Sep 17 18:00:53.764626 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:53.764602 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-dfa2b3\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:53.785059 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:53.785004 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7603-5266f4" podStartSLOduration=1.784989822 podStartE2EDuration="1.784989822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:53.784241411 +0000 UTC m=+1790.346406265" watchObservedRunningTime="2026-09-17 18:00:53.784989822 +0000 UTC m=+1790.347154681" Sep 17 18:00:53.786548 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:53.786524 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-dfa2b3\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:54.147432 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:54.147393 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-dfa2b3\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:54.402543 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:54.402457 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-5266f4"] Sep 17 18:00:54.406188 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:54.406162 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-5266f4"] Sep 17 18:00:54.765653 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:54.765581 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7603-5266f4" container="test-curl-7603-5266f4" Sep 17 18:00:55.770810 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:55.770775 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7603-5266f4" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" containerName="test-curl-7603-5266f4" containerID="cri-o://61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a" gracePeriod=2 Sep 17 18:00:55.773435 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:55.773394 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-5266f4\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:55.926011 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:55.925939 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:55.928639 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:55.928618 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-5266f4\" is forbidden: User \"system:node:ip-10-0-142-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-117.ec2.internal' and this object" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:55.975130 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:55.975107 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxmbp\" (UniqueName: \"kubernetes.io/projected/d646c80a-96bc-4c59-806d-033f8bacbfed-kube-api-access-dxmbp\") pod \"d646c80a-96bc-4c59-806d-033f8bacbfed\" (UID: \"d646c80a-96bc-4c59-806d-033f8bacbfed\") " Sep 17 18:00:55.977042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:55.977012 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d646c80a-96bc-4c59-806d-033f8bacbfed-kube-api-access-dxmbp" (OuterVolumeSpecName: "kube-api-access-dxmbp") pod "d646c80a-96bc-4c59-806d-033f8bacbfed" (UID: "d646c80a-96bc-4c59-806d-033f8bacbfed"). InnerVolumeSpecName "kube-api-access-dxmbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:00:56.076046 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.076022 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxmbp\" (UniqueName: \"kubernetes.io/projected/d646c80a-96bc-4c59-806d-033f8bacbfed-kube-api-access-dxmbp\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:00:56.150061 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.150027 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d646c80a-96bc-4c59-806d-033f8bacbfed" path="/var/lib/kubelet/pods/d646c80a-96bc-4c59-806d-033f8bacbfed/volumes" Sep 17 18:00:56.418453 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.418354 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr"] Sep 17 18:00:56.777751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.777661 2566 generic.go:356] "Generic (PLEG): container finished" podID="d646c80a-96bc-4c59-806d-033f8bacbfed" containerID="648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3" exitCode=137 Sep 17 18:00:56.777751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.777715 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-dfa2b3" Sep 17 18:00:56.777751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.777743 2566 scope.go:117] "RemoveContainer" containerID="648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3" Sep 17 18:00:56.786656 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.786641 2566 scope.go:117] "RemoveContainer" containerID="648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3" Sep 17 18:00:56.786944 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:56.786924 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3\": container with ID starting with 648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3 not found: ID does not exist" containerID="648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3" Sep 17 18:00:56.787009 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:56.786971 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3"} err="failed to get container status \"648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3\": rpc error: code = NotFound desc = could not find container \"648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3\": container with ID starting with 648585963cb8be910d7220460bae030d212a0f02df2a086dfec018c3ba0205e3 not found: ID does not exist" Sep 17 18:00:57.262355 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:57.262317 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml"] Sep 17 18:00:57.954926 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:57.954901 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:58.093667 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.093628 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5jf2\" (UniqueName: \"kubernetes.io/projected/5db02186-9c54-4578-bb93-07a14a8287f8-kube-api-access-c5jf2\") pod \"5db02186-9c54-4578-bb93-07a14a8287f8\" (UID: \"5db02186-9c54-4578-bb93-07a14a8287f8\") " Sep 17 18:00:58.095666 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.095639 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db02186-9c54-4578-bb93-07a14a8287f8-kube-api-access-c5jf2" (OuterVolumeSpecName: "kube-api-access-c5jf2") pod "5db02186-9c54-4578-bb93-07a14a8287f8" (UID: "5db02186-9c54-4578-bb93-07a14a8287f8"). InnerVolumeSpecName "kube-api-access-c5jf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:00:58.149946 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.149918 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" path="/var/lib/kubelet/pods/5db02186-9c54-4578-bb93-07a14a8287f8/volumes" Sep 17 18:00:58.194883 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.194859 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jf2\" (UniqueName: \"kubernetes.io/projected/5db02186-9c54-4578-bb93-07a14a8287f8-kube-api-access-c5jf2\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:00:58.789390 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.789359 2566 generic.go:356] "Generic (PLEG): container finished" podID="5db02186-9c54-4578-bb93-07a14a8287f8" containerID="61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a" exitCode=137 Sep 17 18:00:58.789546 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.789400 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-5266f4" Sep 17 18:00:58.789546 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.789464 2566 scope.go:117] "RemoveContainer" containerID="61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a" Sep 17 18:00:58.797526 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.797500 2566 scope.go:117] "RemoveContainer" containerID="61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a" Sep 17 18:00:58.797788 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:00:58.797770 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a\": container with ID starting with 61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a not found: ID does not exist" containerID="61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a" Sep 17 18:00:58.797841 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:00:58.797800 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a"} err="failed to get container status \"61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a\": rpc error: code = NotFound desc = could not find container \"61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a\": container with ID starting with 61bf2b5279f43515014f746b609436319aa58dbc77e3329f1d6f335dd6f1674a not found: ID does not exist" Sep 17 18:01:11.437159 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.437111 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" podUID="7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" containerName="main" containerID="cri-o://a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793" gracePeriod=45 Sep 17 18:01:11.682729 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.682702 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 18:01:11.826566 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826538 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-dshm\") pod \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " Sep 17 18:01:11.826722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826588 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlg7q\" (UniqueName: \"kubernetes.io/projected/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-kube-api-access-wlg7q\") pod \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " Sep 17 18:01:11.826722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826617 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-home\") pod \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " Sep 17 18:01:11.826722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826634 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tls-certs\") pod \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " Sep 17 18:01:11.826722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826689 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tmp-dir\") pod \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " Sep 17 18:01:11.826920 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826749 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-model-cache\") pod \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\" (UID: \"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669\") " Sep 17 18:01:11.826999 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.826912 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-home" (OuterVolumeSpecName: "home") pod "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" (UID: "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:11.827067 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.827017 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" (UID: "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:11.827150 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.827120 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" (UID: "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:11.828701 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.828674 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-dshm" (OuterVolumeSpecName: "dshm") pod "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" (UID: "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:11.828701 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.828688 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-kube-api-access-wlg7q" (OuterVolumeSpecName: "kube-api-access-wlg7q") pod "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" (UID: "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669"). InnerVolumeSpecName "kube-api-access-wlg7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:01:11.828845 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.828755 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" (UID: "7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:01:11.852507 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.852482 2566 generic.go:356] "Generic (PLEG): container finished" podID="7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" containerID="a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793" exitCode=0 Sep 17 18:01:11.852612 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.852521 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" event={"ID":"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669","Type":"ContainerDied","Data":"a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793"} Sep 17 18:01:11.852612 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.852541 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" Sep 17 18:01:11.852612 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.852558 2566 scope.go:117] "RemoveContainer" containerID="a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793" Sep 17 18:01:11.852778 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.852548 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr" event={"ID":"7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669","Type":"ContainerDied","Data":"5affe9ebc849ad088846be07502c998a33bfa477e7263ffd94bcdc614f2b250f"} Sep 17 18:01:11.861254 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.861237 2566 scope.go:117] "RemoveContainer" containerID="a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793" Sep 17 18:01:11.861526 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:01:11.861509 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793\": container with ID starting with a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793 not found: ID does not exist" containerID="a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793" Sep 17 18:01:11.861577 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.861534 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793"} err="failed to get container status \"a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793\": rpc error: code = NotFound desc = could not find container \"a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793\": container with ID starting with a4f62aef49ff98e25baf09f3411c8b7dabe9d72deec375a8cc2876d8f070f793 not found: ID does not exist" Sep 17 18:01:11.876526 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.876503 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr"] Sep 17 18:01:11.883562 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.883541 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-abb363/sub-test-model-abb363-kserve-75db759d49-zgsrr"] Sep 17 18:01:11.928089 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.928068 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlg7q\" (UniqueName: \"kubernetes.io/projected/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-kube-api-access-wlg7q\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:11.928089 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.928086 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:11.928218 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.928096 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:11.928218 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.928105 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:11.928218 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.928113 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:11.928218 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:11.928120 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.150513 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.150420 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" path="/var/lib/kubelet/pods/7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669/volumes" Sep 17 18:01:12.283950 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.283910 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" podUID="bf0cd405-8bd1-4599-9078-516b75d3c77b" containerName="main" containerID="cri-o://ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4" gracePeriod=45 Sep 17 18:01:12.527480 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.527457 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:01:12.635054 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635016 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-model-cache\") pod \"bf0cd405-8bd1-4599-9078-516b75d3c77b\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " Sep 17 18:01:12.635054 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635056 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-tmp-dir\") pod \"bf0cd405-8bd1-4599-9078-516b75d3c77b\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " Sep 17 18:01:12.635317 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635095 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-dshm\") pod \"bf0cd405-8bd1-4599-9078-516b75d3c77b\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " Sep 17 18:01:12.635317 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635132 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r2v2\" (UniqueName: \"kubernetes.io/projected/bf0cd405-8bd1-4599-9078-516b75d3c77b-kube-api-access-5r2v2\") pod \"bf0cd405-8bd1-4599-9078-516b75d3c77b\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " Sep 17 18:01:12.635317 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635157 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0cd405-8bd1-4599-9078-516b75d3c77b-tls-certs\") pod \"bf0cd405-8bd1-4599-9078-516b75d3c77b\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " Sep 17 18:01:12.635317 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635208 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-home\") pod \"bf0cd405-8bd1-4599-9078-516b75d3c77b\" (UID: \"bf0cd405-8bd1-4599-9078-516b75d3c77b\") " Sep 17 18:01:12.635535 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635333 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf0cd405-8bd1-4599-9078-516b75d3c77b" (UID: "bf0cd405-8bd1-4599-9078-516b75d3c77b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:12.635535 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635383 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf0cd405-8bd1-4599-9078-516b75d3c77b" (UID: "bf0cd405-8bd1-4599-9078-516b75d3c77b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:12.635535 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635525 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-home" (OuterVolumeSpecName: "home") pod "bf0cd405-8bd1-4599-9078-516b75d3c77b" (UID: "bf0cd405-8bd1-4599-9078-516b75d3c77b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:12.635643 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635590 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.635643 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.635611 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.637349 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.637325 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-dshm" (OuterVolumeSpecName: "dshm") pod "bf0cd405-8bd1-4599-9078-516b75d3c77b" (UID: "bf0cd405-8bd1-4599-9078-516b75d3c77b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:01:12.637455 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.637361 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0cd405-8bd1-4599-9078-516b75d3c77b-kube-api-access-5r2v2" (OuterVolumeSpecName: "kube-api-access-5r2v2") pod "bf0cd405-8bd1-4599-9078-516b75d3c77b" (UID: "bf0cd405-8bd1-4599-9078-516b75d3c77b"). InnerVolumeSpecName "kube-api-access-5r2v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:01:12.637455 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.637391 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0cd405-8bd1-4599-9078-516b75d3c77b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf0cd405-8bd1-4599-9078-516b75d3c77b" (UID: "bf0cd405-8bd1-4599-9078-516b75d3c77b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:01:12.736666 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.736575 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.736666 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.736606 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2v2\" (UniqueName: \"kubernetes.io/projected/bf0cd405-8bd1-4599-9078-516b75d3c77b-kube-api-access-5r2v2\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.736666 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.736616 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0cd405-8bd1-4599-9078-516b75d3c77b-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.736666 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.736624 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0cd405-8bd1-4599-9078-516b75d3c77b-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:01:12.859604 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.859567 2566 generic.go:356] "Generic (PLEG): container finished" podID="bf0cd405-8bd1-4599-9078-516b75d3c77b" containerID="ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4" exitCode=0 Sep 17 18:01:12.859604 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.859607 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" Sep 17 18:01:12.859826 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.859614 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" event={"ID":"bf0cd405-8bd1-4599-9078-516b75d3c77b","Type":"ContainerDied","Data":"ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4"} Sep 17 18:01:12.859826 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.859645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml" event={"ID":"bf0cd405-8bd1-4599-9078-516b75d3c77b","Type":"ContainerDied","Data":"48982abd6a2ac6a21d61b7fae3c961adba18e9009885a06b972411f052b983b6"} Sep 17 18:01:12.859826 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.859661 2566 scope.go:117] "RemoveContainer" containerID="ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4" Sep 17 18:01:12.868727 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.868708 2566 scope.go:117] "RemoveContainer" containerID="ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4" Sep 17 18:01:12.869034 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:01:12.869016 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4\": container with ID starting with ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4 not found: ID does not exist" containerID="ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4" Sep 17 18:01:12.869106 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.869041 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4"} err="failed to get container status \"ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4\": rpc error: code = NotFound desc = could not find container \"ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4\": container with ID starting with ceef8a01a40cd5fdb2e6ba6e506fd5a172c3d4ccaeb670ebe838e5c431275aa4 not found: ID does not exist" Sep 17 18:01:12.881183 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.881156 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml"] Sep 17 18:01:12.887086 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:12.887066 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/sub-test-model-7b7c8c-kserve-5f79f576dc-nr5ml"] Sep 17 18:01:14.150289 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:14.150253 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0cd405-8bd1-4599-9078-516b75d3c77b" path="/var/lib/kubelet/pods/bf0cd405-8bd1-4599-9078-516b75d3c77b/volumes" Sep 17 18:01:46.941255 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941219 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l"] Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941546 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" containerName="test-curl-7603-5266f4" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941557 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" containerName="test-curl-7603-5266f4" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941572 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" containerName="main" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941577 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" containerName="main" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941594 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0cd405-8bd1-4599-9078-516b75d3c77b" containerName="main" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941599 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0cd405-8bd1-4599-9078-516b75d3c77b" containerName="main" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941654 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0cd405-8bd1-4599-9078-516b75d3c77b" containerName="main" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941664 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3b5a3b-7b98-45a8-934b-cb3b6c5ed669" containerName="main" Sep 17 18:01:46.941689 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.941670 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db02186-9c54-4578-bb93-07a14a8287f8" containerName="test-curl-7603-5266f4" Sep 17 18:01:46.944426 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.944409 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:46.946922 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.946901 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"rate-test-model-7b7c8c-kserve-self-signed-certs\"" Sep 17 18:01:46.947917 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.947898 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"kube-root-ca.crt\"" Sep 17 18:01:46.948022 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.947976 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"openshift-service-ca.crt\"" Sep 17 18:01:46.948091 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.948039 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7b7c8c\"/\"default-dockercfg-n6kdw\"" Sep 17 18:01:46.956179 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:46.956143 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l"] Sep 17 18:01:47.010288 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.010255 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-home\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.010460 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.010412 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d91301-6735-40f8-a9c7-d9835eff3dca-tls-certs\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.010532 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.010477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-tmp-dir\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.010598 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.010533 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvndg\" (UniqueName: \"kubernetes.io/projected/06d91301-6735-40f8-a9c7-d9835eff3dca-kube-api-access-nvndg\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.010684 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.010644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-dshm\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.010745 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.010698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-model-cache\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111309 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-dshm\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111309 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-model-cache\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111593 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111342 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-home\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111593 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111367 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d91301-6735-40f8-a9c7-d9835eff3dca-tls-certs\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111593 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-tmp-dir\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111593 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111405 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvndg\" (UniqueName: \"kubernetes.io/projected/06d91301-6735-40f8-a9c7-d9835eff3dca-kube-api-access-nvndg\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111820 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111794 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-model-cache\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111885 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111845 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-home\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.111885 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.111868 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-tmp-dir\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.113648 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.113626 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-dshm\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.113866 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.113848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d91301-6735-40f8-a9c7-d9835eff3dca-tls-certs\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.124325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.124305 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvndg\" (UniqueName: \"kubernetes.io/projected/06d91301-6735-40f8-a9c7-d9835eff3dca-kube-api-access-nvndg\") pod \"rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.253767 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.253679 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:47.528365 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:47.528282 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l"] Sep 17 18:01:47.531364 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:01:47.531337 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06d91301_6735_40f8_a9c7_d9835eff3dca.slice/crio-ebf1cabb5823795a9dc616c521a253fb32dba2be54afffb7b3b8d8bd0abc0fa7 WatchSource:0}: Error finding container ebf1cabb5823795a9dc616c521a253fb32dba2be54afffb7b3b8d8bd0abc0fa7: Status 404 returned error can't find the container with id ebf1cabb5823795a9dc616c521a253fb32dba2be54afffb7b3b8d8bd0abc0fa7 Sep 17 18:01:48.031212 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:48.031178 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" event={"ID":"06d91301-6735-40f8-a9c7-d9835eff3dca","Type":"ContainerStarted","Data":"4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a"} Sep 17 18:01:48.031212 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:48.031218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" event={"ID":"06d91301-6735-40f8-a9c7-d9835eff3dca","Type":"ContainerStarted","Data":"ebf1cabb5823795a9dc616c521a253fb32dba2be54afffb7b3b8d8bd0abc0fa7"} Sep 17 18:01:48.051168 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:48.051125 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" podStartSLOduration=2.051109693 podStartE2EDuration="2.051109693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:48.049920912 +0000 UTC m=+1844.612085773" watchObservedRunningTime="2026-09-17 18:01:48.051109693 +0000 UTC m=+1844.613274581" Sep 17 18:01:57.254130 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:57.254094 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:57.254542 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:57.254376 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:57.266768 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:57.266740 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:01:58.092618 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:01:58.092588 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:02:25.742224 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:25.742187 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l"] Sep 17 18:02:32.027121 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.027069 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph"] Sep 17 18:02:32.031186 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.031163 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.033737 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.033714 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-dba35d-serving-cert\"" Sep 17 18:02:32.042262 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.042239 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph"] Sep 17 18:02:32.099884 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.099856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-metrics-certs\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.099884 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.099887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1961a2-2fe0-4d81-a355-6d7610950e29-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.100136 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.099916 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hq2f\" (UniqueName: \"kubernetes.io/projected/1a1961a2-2fe0-4d81-a355-6d7610950e29-kube-api-access-8hq2f\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.100136 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.100049 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.200636 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.200604 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-metrics-certs\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.200818 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.200644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1961a2-2fe0-4d81-a355-6d7610950e29-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.200818 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.200688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hq2f\" (UniqueName: \"kubernetes.io/projected/1a1961a2-2fe0-4d81-a355-6d7610950e29-kube-api-access-8hq2f\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.200818 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.200747 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.201032 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:02:32.200866 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-serving-cert: secret "maas-api-e2e-ipp-a-dba35d-serving-cert" not found Sep 17 18:02:32.201032 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:02:32.200945 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls podName:1a1961a2-2fe0-4d81-a355-6d7610950e29 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:32.70092212 +0000 UTC m=+1889.263086972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls") pod "maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" (UID: "1a1961a2-2fe0-4d81-a355-6d7610950e29") : secret "maas-api-e2e-ipp-a-dba35d-serving-cert" not found Sep 17 18:02:32.201450 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.201424 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1961a2-2fe0-4d81-a355-6d7610950e29-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.203088 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.203067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-metrics-certs\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.211969 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.211933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hq2f\" (UniqueName: \"kubernetes.io/projected/1a1961a2-2fe0-4d81-a355-6d7610950e29-kube-api-access-8hq2f\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.705454 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.705411 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.707903 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.707867 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:32.741323 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.741290 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn"] Sep 17 18:02:32.744692 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.744675 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.747347 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.747326 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dba35d\"" Sep 17 18:02:32.751326 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.751303 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr"] Sep 17 18:02:32.754754 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.754735 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.764864 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.764843 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn"] Sep 17 18:02:32.768053 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.768022 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr"] Sep 17 18:02:32.805871 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.805840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529e660a-76a9-40a8-b464-fc5bde5e6040-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.806026 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.805882 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j96tp\" (UniqueName: \"kubernetes.io/projected/72ae4724-ae42-4baa-9a16-e16b664d0597-kube-api-access-j96tp\") pod \"payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.806026 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.805912 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmt6\" (UniqueName: \"kubernetes.io/projected/529e660a-76a9-40a8-b464-fc5bde5e6040-kube-api-access-2cmt6\") pod \"payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.806026 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.806004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae4724-ae42-4baa-9a16-e16b664d0597-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.907235 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.907196 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae4724-ae42-4baa-9a16-e16b664d0597-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.907418 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.907284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529e660a-76a9-40a8-b464-fc5bde5e6040-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.907418 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.907319 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j96tp\" (UniqueName: \"kubernetes.io/projected/72ae4724-ae42-4baa-9a16-e16b664d0597-kube-api-access-j96tp\") pod \"payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.907563 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.907459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmt6\" (UniqueName: \"kubernetes.io/projected/529e660a-76a9-40a8-b464-fc5bde5e6040-kube-api-access-2cmt6\") pod \"payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.908037 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.908013 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae4724-ae42-4baa-9a16-e16b664d0597-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.908126 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.908060 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529e660a-76a9-40a8-b464-fc5bde5e6040-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.919685 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.919655 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmt6\" (UniqueName: \"kubernetes.io/projected/529e660a-76a9-40a8-b464-fc5bde5e6040-kube-api-access-2cmt6\") pod \"payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:32.919892 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.919871 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j96tp\" (UniqueName: \"kubernetes.io/projected/72ae4724-ae42-4baa-9a16-e16b664d0597-kube-api-access-j96tp\") pod \"payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:32.943330 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:32.943305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:33.055684 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.055651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:33.065296 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.065230 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:33.222237 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.222105 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph"] Sep 17 18:02:33.224919 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:02:33.224889 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1961a2_2fe0_4d81_a355_6d7610950e29.slice/crio-abd9db67d6d9010a75f2a361a8e6496e02a48f038919c32e940f9621b888e027 WatchSource:0}: Error finding container abd9db67d6d9010a75f2a361a8e6496e02a48f038919c32e940f9621b888e027: Status 404 returned error can't find the container with id abd9db67d6d9010a75f2a361a8e6496e02a48f038919c32e940f9621b888e027 Sep 17 18:02:33.226285 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.226266 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:02:33.254405 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.254381 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" event={"ID":"1a1961a2-2fe0-4d81-a355-6d7610950e29","Type":"ContainerStarted","Data":"abd9db67d6d9010a75f2a361a8e6496e02a48f038919c32e940f9621b888e027"} Sep 17 18:02:33.340263 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.340238 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn"] Sep 17 18:02:33.342421 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:02:33.342367 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ae4724_ae42_4baa_9a16_e16b664d0597.slice/crio-edca48c1889897f642b284edb6bac2a43012e7f33f8f5f994a58b64183edc4c5 WatchSource:0}: Error finding container edca48c1889897f642b284edb6bac2a43012e7f33f8f5f994a58b64183edc4c5: Status 404 returned error can't find the container with id edca48c1889897f642b284edb6bac2a43012e7f33f8f5f994a58b64183edc4c5 Sep 17 18:02:33.348208 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:33.348179 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr"] Sep 17 18:02:33.348749 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:02:33.348726 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529e660a_76a9_40a8_b464_fc5bde5e6040.slice/crio-a4ef4a428c2985736a7c039b6915b774758faf9dd3effdae4a3c0d15517eb186 WatchSource:0}: Error finding container a4ef4a428c2985736a7c039b6915b774758faf9dd3effdae4a3c0d15517eb186: Status 404 returned error can't find the container with id a4ef4a428c2985736a7c039b6915b774758faf9dd3effdae4a3c0d15517eb186 Sep 17 18:02:34.261255 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.261203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" event={"ID":"529e660a-76a9-40a8-b464-fc5bde5e6040","Type":"ContainerStarted","Data":"0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120"} Sep 17 18:02:34.261662 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.261271 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" event={"ID":"529e660a-76a9-40a8-b464-fc5bde5e6040","Type":"ContainerStarted","Data":"a4ef4a428c2985736a7c039b6915b774758faf9dd3effdae4a3c0d15517eb186"} Sep 17 18:02:34.261662 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.261319 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:02:34.263118 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.263078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" event={"ID":"72ae4724-ae42-4baa-9a16-e16b664d0597","Type":"ContainerStarted","Data":"e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a"} Sep 17 18:02:34.263269 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.263134 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:34.263269 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.263156 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" event={"ID":"72ae4724-ae42-4baa-9a16-e16b664d0597","Type":"ContainerStarted","Data":"edca48c1889897f642b284edb6bac2a43012e7f33f8f5f994a58b64183edc4c5"} Sep 17 18:02:34.264793 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.264763 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" event={"ID":"1a1961a2-2fe0-4d81-a355-6d7610950e29","Type":"ContainerStarted","Data":"8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb"} Sep 17 18:02:34.264903 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.264812 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:34.279522 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.279470 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" podStartSLOduration=2.279448359 podStartE2EDuration="2.279448359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:34.278451662 +0000 UTC m=+1890.840616518" watchObservedRunningTime="2026-09-17 18:02:34.279448359 +0000 UTC m=+1890.841613224" Sep 17 18:02:34.307299 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.307255 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" podStartSLOduration=1.80875786 podStartE2EDuration="2.307241694s" totalImagesPullingTime="498.483834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:34.306027761 +0000 UTC m=+1890.868192618" watchObservedRunningTime="2026-09-17 18:02:34.307241694 +0000 UTC m=+1890.869406554" Sep 17 18:02:34.332433 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:34.332389 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" podStartSLOduration=2.332373442 podStartE2EDuration="2.332373442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:34.3316693 +0000 UTC m=+1890.893834162" watchObservedRunningTime="2026-09-17 18:02:34.332373442 +0000 UTC m=+1890.894538303" Sep 17 18:02:40.276390 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:40.276363 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:02:40.761429 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:40.761371 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" podUID="06d91301-6735-40f8-a9c7-d9835eff3dca" containerName="main" containerID="cri-o://4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a" gracePeriod=45 Sep 17 18:02:41.000885 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.000861 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:02:41.081690 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.081652 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-dshm\") pod \"06d91301-6735-40f8-a9c7-d9835eff3dca\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " Sep 17 18:02:41.081860 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.081730 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-model-cache\") pod \"06d91301-6735-40f8-a9c7-d9835eff3dca\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " Sep 17 18:02:41.081860 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.081772 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d91301-6735-40f8-a9c7-d9835eff3dca-tls-certs\") pod \"06d91301-6735-40f8-a9c7-d9835eff3dca\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " Sep 17 18:02:41.081860 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.081796 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvndg\" (UniqueName: \"kubernetes.io/projected/06d91301-6735-40f8-a9c7-d9835eff3dca-kube-api-access-nvndg\") pod \"06d91301-6735-40f8-a9c7-d9835eff3dca\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " Sep 17 18:02:41.081860 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.081819 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-tmp-dir\") pod \"06d91301-6735-40f8-a9c7-d9835eff3dca\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " Sep 17 18:02:41.081860 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.081838 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-home\") pod \"06d91301-6735-40f8-a9c7-d9835eff3dca\" (UID: \"06d91301-6735-40f8-a9c7-d9835eff3dca\") " Sep 17 18:02:41.082165 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.082079 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06d91301-6735-40f8-a9c7-d9835eff3dca" (UID: "06d91301-6735-40f8-a9c7-d9835eff3dca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:02:41.082165 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.082091 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-model-cache" (OuterVolumeSpecName: "model-cache") pod "06d91301-6735-40f8-a9c7-d9835eff3dca" (UID: "06d91301-6735-40f8-a9c7-d9835eff3dca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:02:41.082283 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.082260 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-home" (OuterVolumeSpecName: "home") pod "06d91301-6735-40f8-a9c7-d9835eff3dca" (UID: "06d91301-6735-40f8-a9c7-d9835eff3dca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:02:41.083839 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.083813 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-dshm" (OuterVolumeSpecName: "dshm") pod "06d91301-6735-40f8-a9c7-d9835eff3dca" (UID: "06d91301-6735-40f8-a9c7-d9835eff3dca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:02:41.083969 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.083873 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d91301-6735-40f8-a9c7-d9835eff3dca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06d91301-6735-40f8-a9c7-d9835eff3dca" (UID: "06d91301-6735-40f8-a9c7-d9835eff3dca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:02:41.084030 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.083979 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d91301-6735-40f8-a9c7-d9835eff3dca-kube-api-access-nvndg" (OuterVolumeSpecName: "kube-api-access-nvndg") pod "06d91301-6735-40f8-a9c7-d9835eff3dca" (UID: "06d91301-6735-40f8-a9c7-d9835eff3dca"). InnerVolumeSpecName "kube-api-access-nvndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:02:41.183226 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.183198 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvndg\" (UniqueName: \"kubernetes.io/projected/06d91301-6735-40f8-a9c7-d9835eff3dca-kube-api-access-nvndg\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:02:41.183226 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.183225 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:02:41.183372 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.183236 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:02:41.183372 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.183244 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:02:41.183372 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.183251 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d91301-6735-40f8-a9c7-d9835eff3dca-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:02:41.183372 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.183259 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d91301-6735-40f8-a9c7-d9835eff3dca-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:02:41.301828 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.301799 2566 generic.go:356] "Generic (PLEG): container finished" podID="06d91301-6735-40f8-a9c7-d9835eff3dca" containerID="4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a" exitCode=0 Sep 17 18:02:41.302192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.301837 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" event={"ID":"06d91301-6735-40f8-a9c7-d9835eff3dca","Type":"ContainerDied","Data":"4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a"} Sep 17 18:02:41.302192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.301841 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" Sep 17 18:02:41.302192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.301866 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l" event={"ID":"06d91301-6735-40f8-a9c7-d9835eff3dca","Type":"ContainerDied","Data":"ebf1cabb5823795a9dc616c521a253fb32dba2be54afffb7b3b8d8bd0abc0fa7"} Sep 17 18:02:41.302192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.301882 2566 scope.go:117] "RemoveContainer" containerID="4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a" Sep 17 18:02:41.311408 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.311392 2566 scope.go:117] "RemoveContainer" containerID="4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a" Sep 17 18:02:41.311670 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:02:41.311651 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a\": container with ID starting with 4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a not found: ID does not exist" containerID="4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a" Sep 17 18:02:41.311738 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.311679 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a"} err="failed to get container status \"4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a\": rpc error: code = NotFound desc = could not find container \"4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a\": container with ID starting with 4b49312a94f9da1fedd0f777e3980bba75c0f99f91ee30853c350bf4190e3d8a not found: ID does not exist" Sep 17 18:02:41.325464 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.325441 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l"] Sep 17 18:02:41.334180 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:41.334124 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7b7c8c/rate-test-model-7b7c8c-kserve-5f76fc74cf-qp77l"] Sep 17 18:02:42.150816 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:42.150784 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d91301-6735-40f8-a9c7-d9835eff3dca" path="/var/lib/kubelet/pods/06d91301-6735-40f8-a9c7-d9835eff3dca/volumes" Sep 17 18:02:45.273170 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:45.273125 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:02:45.273652 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:02:45.273598 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:03:04.013364 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.013329 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4"] Sep 17 18:03:04.013842 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.013702 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d91301-6735-40f8-a9c7-d9835eff3dca" containerName="main" Sep 17 18:03:04.013842 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.013721 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d91301-6735-40f8-a9c7-d9835eff3dca" containerName="main" Sep 17 18:03:04.013842 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.013811 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d91301-6735-40f8-a9c7-d9835eff3dca" containerName="main" Sep 17 18:03:04.018084 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.018067 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.032022 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.031997 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b77ab2-dockercfg-5lln4\"" Sep 17 18:03:04.041140 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.041115 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b77ab2\"" Sep 17 18:03:04.045339 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.045312 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4"] Sep 17 18:03:04.182987 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.182938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zzgm\" (UniqueName: \"kubernetes.io/projected/92e315cb-5d40-4bb4-9932-ea1d52086cdf-kube-api-access-9zzgm\") pod \"payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.183163 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.183039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e315cb-5d40-4bb4-9932-ea1d52086cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.283603 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.283508 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e315cb-5d40-4bb4-9932-ea1d52086cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.283603 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.283563 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zzgm\" (UniqueName: \"kubernetes.io/projected/92e315cb-5d40-4bb4-9932-ea1d52086cdf-kube-api-access-9zzgm\") pod \"payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.284322 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.284289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e315cb-5d40-4bb4-9932-ea1d52086cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.316567 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.316539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zzgm\" (UniqueName: \"kubernetes.io/projected/92e315cb-5d40-4bb4-9932-ea1d52086cdf-kube-api-access-9zzgm\") pod \"payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.328463 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.328442 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:04.614163 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:04.614084 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4"] Sep 17 18:03:04.616479 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:03:04.616449 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e315cb_5d40_4bb4_9932_ea1d52086cdf.slice/crio-c64a5fba6861e82a468743a09db21b7c6169a58f1ec15927ba7f2ba07e9ae606 WatchSource:0}: Error finding container c64a5fba6861e82a468743a09db21b7c6169a58f1ec15927ba7f2ba07e9ae606: Status 404 returned error can't find the container with id c64a5fba6861e82a468743a09db21b7c6169a58f1ec15927ba7f2ba07e9ae606 Sep 17 18:03:05.426792 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:05.426748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" event={"ID":"92e315cb-5d40-4bb4-9932-ea1d52086cdf","Type":"ContainerStarted","Data":"08c8feed8b3c3c3bfca47a367b52a911f9f5385f7f55f9ce299315cb3744a5d8"} Sep 17 18:03:05.427232 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:05.426818 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" event={"ID":"92e315cb-5d40-4bb4-9932-ea1d52086cdf","Type":"ContainerStarted","Data":"c64a5fba6861e82a468743a09db21b7c6169a58f1ec15927ba7f2ba07e9ae606"} Sep 17 18:03:05.427232 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:05.426849 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:05.466525 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:05.466471 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" podStartSLOduration=2.4664552840000002 podStartE2EDuration="2.466455284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:05.461552765 +0000 UTC m=+1922.023717626" watchObservedRunningTime="2026-09-17 18:03:05.466455284 +0000 UTC m=+1922.028620143" Sep 17 18:03:16.433515 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:16.433484 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:03:50.103254 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.103217 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms"] Sep 17 18:03:50.111426 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.111387 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.121244 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.121224 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dba35d\"/\"ipp-route-dba35d-kserve-self-signed-certs\"" Sep 17 18:03:50.121357 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.121280 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dba35d\"/\"kube-root-ca.crt\"" Sep 17 18:03:50.122745 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.122728 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dba35d\"/\"default-dockercfg-7csfx\"" Sep 17 18:03:50.122848 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.122830 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dba35d\"/\"openshift-service-ca.crt\"" Sep 17 18:03:50.130406 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.130382 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms"] Sep 17 18:03:50.275700 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.275663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-tmp-dir\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.275882 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.275759 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028ba429-364e-4b8d-85f2-df848cc29b26-tls-certs\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.275882 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.275790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-home\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.275882 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.275822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-model-cache\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.275882 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.275870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-dshm\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.276060 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.275885 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp9fq\" (UniqueName: \"kubernetes.io/projected/028ba429-364e-4b8d-85f2-df848cc29b26-kube-api-access-xp9fq\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377187 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-tmp-dir\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377187 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377150 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028ba429-364e-4b8d-85f2-df848cc29b26-tls-certs\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377187 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-home\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377493 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-model-cache\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377493 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-dshm\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377493 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp9fq\" (UniqueName: \"kubernetes.io/projected/028ba429-364e-4b8d-85f2-df848cc29b26-kube-api-access-xp9fq\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377663 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377590 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-home\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377724 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377666 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-tmp-dir\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.377845 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.377827 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-model-cache\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.379778 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.379753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-dshm\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.380067 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.380045 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028ba429-364e-4b8d-85f2-df848cc29b26-tls-certs\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.391292 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.391266 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp9fq\" (UniqueName: \"kubernetes.io/projected/028ba429-364e-4b8d-85f2-df848cc29b26-kube-api-access-xp9fq\") pod \"ipp-route-dba35d-kserve-6969f9dd8c-bnlms\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.420555 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.420523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:03:50.708060 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:50.708012 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms"] Sep 17 18:03:50.710291 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:03:50.710263 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod028ba429_364e_4b8d_85f2_df848cc29b26.slice/crio-a78e1e00a985095f2feffc4e64a315baca51eb52f746bab8b871434a3261607d WatchSource:0}: Error finding container a78e1e00a985095f2feffc4e64a315baca51eb52f746bab8b871434a3261607d: Status 404 returned error can't find the container with id a78e1e00a985095f2feffc4e64a315baca51eb52f746bab8b871434a3261607d Sep 17 18:03:51.648798 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:51.648763 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" event={"ID":"028ba429-364e-4b8d-85f2-df848cc29b26","Type":"ContainerStarted","Data":"364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768"} Sep 17 18:03:51.649283 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:03:51.648812 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" event={"ID":"028ba429-364e-4b8d-85f2-df848cc29b26","Type":"ContainerStarted","Data":"a78e1e00a985095f2feffc4e64a315baca51eb52f746bab8b871434a3261607d"} Sep 17 18:04:00.421561 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:00.421481 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:04:00.421561 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:00.421532 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:04:00.433753 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:00.433731 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:04:00.466925 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:00.466883 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" podStartSLOduration=10.466870272 podStartE2EDuration="10.466870272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:51.678243605 +0000 UTC m=+1968.240408467" watchObservedRunningTime="2026-09-17 18:04:00.466870272 +0000 UTC m=+1977.029035131" Sep 17 18:04:00.707046 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:00.706948 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:04:21.648548 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:21.648513 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr_529e660a-76a9-40a8-b464-fc5bde5e6040/payload-processing/0.log" Sep 17 18:04:31.284752 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.284721 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr"] Sep 17 18:04:31.288182 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.288164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.292734 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.292715 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-58f131-dockercfg-twl9n\"" Sep 17 18:04:31.292841 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.292740 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-58f131\"" Sep 17 18:04:31.304806 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.304783 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr"] Sep 17 18:04:31.444467 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.444438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a397e13-5928-4850-868d-6c93ab8a3b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.444467 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.444471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2g76\" (UniqueName: \"kubernetes.io/projected/1a397e13-5928-4850-868d-6c93ab8a3b3a-kube-api-access-m2g76\") pod \"payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.545636 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.545557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a397e13-5928-4850-868d-6c93ab8a3b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.545636 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.545588 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2g76\" (UniqueName: \"kubernetes.io/projected/1a397e13-5928-4850-868d-6c93ab8a3b3a-kube-api-access-m2g76\") pod \"payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.546269 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.546249 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a397e13-5928-4850-868d-6c93ab8a3b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.557422 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.557392 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2g76\" (UniqueName: \"kubernetes.io/projected/1a397e13-5928-4850-868d-6c93ab8a3b3a-kube-api-access-m2g76\") pod \"payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.597878 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.597847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:31.881846 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:31.881820 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr"] Sep 17 18:04:31.884004 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:04:31.883971 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a397e13_5928_4850_868d_6c93ab8a3b3a.slice/crio-52d9575feac8ed710cfd3dd8cbeaf8d540a0076520648cd49714f011809bcd65 WatchSource:0}: Error finding container 52d9575feac8ed710cfd3dd8cbeaf8d540a0076520648cd49714f011809bcd65: Status 404 returned error can't find the container with id 52d9575feac8ed710cfd3dd8cbeaf8d540a0076520648cd49714f011809bcd65 Sep 17 18:04:32.859256 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:32.859223 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" event={"ID":"1a397e13-5928-4850-868d-6c93ab8a3b3a","Type":"ContainerStarted","Data":"359b3b59943d1e2dd22da2fef596da19c6a42aa1cdb336955834f401d5f92327"} Sep 17 18:04:32.859709 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:32.859277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" event={"ID":"1a397e13-5928-4850-868d-6c93ab8a3b3a","Type":"ContainerStarted","Data":"52d9575feac8ed710cfd3dd8cbeaf8d540a0076520648cd49714f011809bcd65"} Sep 17 18:04:32.859709 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:32.859298 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:04:32.888451 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:32.888393 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" podStartSLOduration=1.888372535 podStartE2EDuration="1.888372535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:32.884854381 +0000 UTC m=+2009.447019242" watchObservedRunningTime="2026-09-17 18:04:32.888372535 +0000 UTC m=+2009.450537397" Sep 17 18:04:43.866604 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:04:43.866570 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:05:08.883435 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:08.883398 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr"] Sep 17 18:05:08.884005 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:08.883615 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" podUID="1a397e13-5928-4850-868d-6c93ab8a3b3a" containerName="payload-pre-processing" containerID="cri-o://359b3b59943d1e2dd22da2fef596da19c6a42aa1cdb336955834f401d5f92327" gracePeriod=30 Sep 17 18:05:09.042184 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.042149 2566 generic.go:356] "Generic (PLEG): container finished" podID="1a397e13-5928-4850-868d-6c93ab8a3b3a" containerID="359b3b59943d1e2dd22da2fef596da19c6a42aa1cdb336955834f401d5f92327" exitCode=0 Sep 17 18:05:09.042367 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.042223 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" event={"ID":"1a397e13-5928-4850-868d-6c93ab8a3b3a","Type":"ContainerDied","Data":"359b3b59943d1e2dd22da2fef596da19c6a42aa1cdb336955834f401d5f92327"} Sep 17 18:05:09.125644 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.125620 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:05:09.176107 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.176019 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2g76\" (UniqueName: \"kubernetes.io/projected/1a397e13-5928-4850-868d-6c93ab8a3b3a-kube-api-access-m2g76\") pod \"1a397e13-5928-4850-868d-6c93ab8a3b3a\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " Sep 17 18:05:09.176107 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.176076 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a397e13-5928-4850-868d-6c93ab8a3b3a-plugins-config-volume\") pod \"1a397e13-5928-4850-868d-6c93ab8a3b3a\" (UID: \"1a397e13-5928-4850-868d-6c93ab8a3b3a\") " Sep 17 18:05:09.176728 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.176666 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a397e13-5928-4850-868d-6c93ab8a3b3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a397e13-5928-4850-868d-6c93ab8a3b3a" (UID: "1a397e13-5928-4850-868d-6c93ab8a3b3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:09.178857 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.178829 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a397e13-5928-4850-868d-6c93ab8a3b3a-kube-api-access-m2g76" (OuterVolumeSpecName: "kube-api-access-m2g76") pod "1a397e13-5928-4850-868d-6c93ab8a3b3a" (UID: "1a397e13-5928-4850-868d-6c93ab8a3b3a"). InnerVolumeSpecName "kube-api-access-m2g76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:09.277673 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.277643 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2g76\" (UniqueName: \"kubernetes.io/projected/1a397e13-5928-4850-868d-6c93ab8a3b3a-kube-api-access-m2g76\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:09.277673 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:09.277668 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a397e13-5928-4850-868d-6c93ab8a3b3a-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:10.049576 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:10.049534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" event={"ID":"1a397e13-5928-4850-868d-6c93ab8a3b3a","Type":"ContainerDied","Data":"52d9575feac8ed710cfd3dd8cbeaf8d540a0076520648cd49714f011809bcd65"} Sep 17 18:05:10.050079 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:10.049600 2566 scope.go:117] "RemoveContainer" containerID="359b3b59943d1e2dd22da2fef596da19c6a42aa1cdb336955834f401d5f92327" Sep 17 18:05:10.050079 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:10.049555 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr" Sep 17 18:05:10.102551 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:10.102516 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr"] Sep 17 18:05:10.112453 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:10.112430 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-58f131-bc8f9d86b-9vfnr"] Sep 17 18:05:10.149772 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:10.149741 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a397e13-5928-4850-868d-6c93ab8a3b3a" path="/var/lib/kubelet/pods/1a397e13-5928-4850-868d-6c93ab8a3b3a/volumes" Sep 17 18:05:37.610298 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.610261 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph"] Sep 17 18:05:37.610784 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.610474 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" podUID="1a1961a2-2fe0-4d81-a355-6d7610950e29" containerName="maas-api" containerID="cri-o://8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb" gracePeriod=30 Sep 17 18:05:37.709944 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.709909 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr"] Sep 17 18:05:37.710248 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.710191 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" podUID="529e660a-76a9-40a8-b464-fc5bde5e6040" containerName="payload-processing" containerID="cri-o://0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120" gracePeriod=30 Sep 17 18:05:37.733203 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.733172 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn"] Sep 17 18:05:37.733451 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.733425 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" podUID="72ae4724-ae42-4baa-9a16-e16b664d0597" containerName="payload-pre-processing" containerID="cri-o://e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a" gracePeriod=30 Sep 17 18:05:37.883441 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.883408 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:05:37.938337 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.938310 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hq2f\" (UniqueName: \"kubernetes.io/projected/1a1961a2-2fe0-4d81-a355-6d7610950e29-kube-api-access-8hq2f\") pod \"1a1961a2-2fe0-4d81-a355-6d7610950e29\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " Sep 17 18:05:37.938455 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.938393 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-metrics-certs\") pod \"1a1961a2-2fe0-4d81-a355-6d7610950e29\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " Sep 17 18:05:37.938455 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.938440 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls\") pod \"1a1961a2-2fe0-4d81-a355-6d7610950e29\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " Sep 17 18:05:37.938593 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.938467 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1961a2-2fe0-4d81-a355-6d7610950e29-openshift-service-ca\") pod \"1a1961a2-2fe0-4d81-a355-6d7610950e29\" (UID: \"1a1961a2-2fe0-4d81-a355-6d7610950e29\") " Sep 17 18:05:37.939112 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.939079 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1961a2-2fe0-4d81-a355-6d7610950e29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a1961a2-2fe0-4d81-a355-6d7610950e29" (UID: "1a1961a2-2fe0-4d81-a355-6d7610950e29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:37.940581 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.940558 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1961a2-2fe0-4d81-a355-6d7610950e29-kube-api-access-8hq2f" (OuterVolumeSpecName: "kube-api-access-8hq2f") pod "1a1961a2-2fe0-4d81-a355-6d7610950e29" (UID: "1a1961a2-2fe0-4d81-a355-6d7610950e29"). InnerVolumeSpecName "kube-api-access-8hq2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:37.940673 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.940595 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a1961a2-2fe0-4d81-a355-6d7610950e29" (UID: "1a1961a2-2fe0-4d81-a355-6d7610950e29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:37.941001 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.940930 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a1961a2-2fe0-4d81-a355-6d7610950e29" (UID: "1a1961a2-2fe0-4d81-a355-6d7610950e29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:37.998825 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:37.998800 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:05:38.004722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.004698 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:05:38.039316 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039289 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cmt6\" (UniqueName: \"kubernetes.io/projected/529e660a-76a9-40a8-b464-fc5bde5e6040-kube-api-access-2cmt6\") pod \"529e660a-76a9-40a8-b464-fc5bde5e6040\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " Sep 17 18:05:38.039469 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039344 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529e660a-76a9-40a8-b464-fc5bde5e6040-plugins-config-volume\") pod \"529e660a-76a9-40a8-b464-fc5bde5e6040\" (UID: \"529e660a-76a9-40a8-b464-fc5bde5e6040\") " Sep 17 18:05:38.039469 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039371 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae4724-ae42-4baa-9a16-e16b664d0597-plugins-config-volume\") pod \"72ae4724-ae42-4baa-9a16-e16b664d0597\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " Sep 17 18:05:38.039469 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039453 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j96tp\" (UniqueName: \"kubernetes.io/projected/72ae4724-ae42-4baa-9a16-e16b664d0597-kube-api-access-j96tp\") pod \"72ae4724-ae42-4baa-9a16-e16b664d0597\" (UID: \"72ae4724-ae42-4baa-9a16-e16b664d0597\") " Sep 17 18:05:38.039888 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039699 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529e660a-76a9-40a8-b464-fc5bde5e6040-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "529e660a-76a9-40a8-b464-fc5bde5e6040" (UID: "529e660a-76a9-40a8-b464-fc5bde5e6040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:38.039888 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039807 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.039888 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039825 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a1961a2-2fe0-4d81-a355-6d7610950e29-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.039888 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039837 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1961a2-2fe0-4d81-a355-6d7610950e29-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.039888 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039850 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529e660a-76a9-40a8-b464-fc5bde5e6040-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.039888 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039865 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hq2f\" (UniqueName: \"kubernetes.io/projected/1a1961a2-2fe0-4d81-a355-6d7610950e29-kube-api-access-8hq2f\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.040327 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.039987 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ae4724-ae42-4baa-9a16-e16b664d0597-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72ae4724-ae42-4baa-9a16-e16b664d0597" (UID: "72ae4724-ae42-4baa-9a16-e16b664d0597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:38.041592 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.041570 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529e660a-76a9-40a8-b464-fc5bde5e6040-kube-api-access-2cmt6" (OuterVolumeSpecName: "kube-api-access-2cmt6") pod "529e660a-76a9-40a8-b464-fc5bde5e6040" (UID: "529e660a-76a9-40a8-b464-fc5bde5e6040"). InnerVolumeSpecName "kube-api-access-2cmt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:38.042119 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.042093 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ae4724-ae42-4baa-9a16-e16b664d0597-kube-api-access-j96tp" (OuterVolumeSpecName: "kube-api-access-j96tp") pod "72ae4724-ae42-4baa-9a16-e16b664d0597" (UID: "72ae4724-ae42-4baa-9a16-e16b664d0597"). InnerVolumeSpecName "kube-api-access-j96tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:38.141144 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.141063 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j96tp\" (UniqueName: \"kubernetes.io/projected/72ae4724-ae42-4baa-9a16-e16b664d0597-kube-api-access-j96tp\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.141144 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.141091 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cmt6\" (UniqueName: \"kubernetes.io/projected/529e660a-76a9-40a8-b464-fc5bde5e6040-kube-api-access-2cmt6\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.141144 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.141105 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae4724-ae42-4baa-9a16-e16b664d0597-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:38.197043 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.197001 2566 generic.go:356] "Generic (PLEG): container finished" podID="529e660a-76a9-40a8-b464-fc5bde5e6040" containerID="0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120" exitCode=0 Sep 17 18:05:38.197196 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.197064 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" Sep 17 18:05:38.197196 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.197091 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" event={"ID":"529e660a-76a9-40a8-b464-fc5bde5e6040","Type":"ContainerDied","Data":"0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120"} Sep 17 18:05:38.197196 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.197147 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr" event={"ID":"529e660a-76a9-40a8-b464-fc5bde5e6040","Type":"ContainerDied","Data":"a4ef4a428c2985736a7c039b6915b774758faf9dd3effdae4a3c0d15517eb186"} Sep 17 18:05:38.197196 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.197170 2566 scope.go:117] "RemoveContainer" containerID="0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120" Sep 17 18:05:38.198450 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.198376 2566 generic.go:356] "Generic (PLEG): container finished" podID="72ae4724-ae42-4baa-9a16-e16b664d0597" containerID="e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a" exitCode=0 Sep 17 18:05:38.198450 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.198429 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" Sep 17 18:05:38.198590 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.198426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" event={"ID":"72ae4724-ae42-4baa-9a16-e16b664d0597","Type":"ContainerDied","Data":"e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a"} Sep 17 18:05:38.198590 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.198558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn" event={"ID":"72ae4724-ae42-4baa-9a16-e16b664d0597","Type":"ContainerDied","Data":"edca48c1889897f642b284edb6bac2a43012e7f33f8f5f994a58b64183edc4c5"} Sep 17 18:05:38.200140 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.200117 2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1961a2-2fe0-4d81-a355-6d7610950e29" containerID="8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb" exitCode=0 Sep 17 18:05:38.200268 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.200152 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" event={"ID":"1a1961a2-2fe0-4d81-a355-6d7610950e29","Type":"ContainerDied","Data":"8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb"} Sep 17 18:05:38.200268 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.200180 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" Sep 17 18:05:38.200268 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.200199 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph" event={"ID":"1a1961a2-2fe0-4d81-a355-6d7610950e29","Type":"ContainerDied","Data":"abd9db67d6d9010a75f2a361a8e6496e02a48f038919c32e940f9621b888e027"} Sep 17 18:05:38.206353 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.206338 2566 scope.go:117] "RemoveContainer" containerID="0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120" Sep 17 18:05:38.206629 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:05:38.206588 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120\": container with ID starting with 0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120 not found: ID does not exist" containerID="0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120" Sep 17 18:05:38.206629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.206612 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120"} err="failed to get container status \"0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120\": rpc error: code = NotFound desc = could not find container \"0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120\": container with ID starting with 0ec956f3bec8231c10bab693c16eed296fbdf0707c65fa1543bd4d17be1af120 not found: ID does not exist" Sep 17 18:05:38.206629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.206629 2566 scope.go:117] "RemoveContainer" containerID="e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a" Sep 17 18:05:38.214140 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.214125 2566 scope.go:117] "RemoveContainer" containerID="e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a" Sep 17 18:05:38.214387 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:05:38.214372 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a\": container with ID starting with e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a not found: ID does not exist" containerID="e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a" Sep 17 18:05:38.214452 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.214391 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a"} err="failed to get container status \"e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a\": rpc error: code = NotFound desc = could not find container \"e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a\": container with ID starting with e2458d250d183c840b8b727732a69ce6298f88b540fffae9de7dcadebf50d82a not found: ID does not exist" Sep 17 18:05:38.214452 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.214407 2566 scope.go:117] "RemoveContainer" containerID="8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb" Sep 17 18:05:38.221816 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.221801 2566 scope.go:117] "RemoveContainer" containerID="8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb" Sep 17 18:05:38.222104 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:05:38.222084 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb\": container with ID starting with 8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb not found: ID does not exist" containerID="8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb" Sep 17 18:05:38.222172 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.222109 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb"} err="failed to get container status \"8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb\": rpc error: code = NotFound desc = could not find container \"8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb\": container with ID starting with 8f22aaad723c47a9ff0be91683cd76e4617e8ebee6a67baeded8cba44eac1dbb not found: ID does not exist" Sep 17 18:05:38.233722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.233701 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr"] Sep 17 18:05:38.238233 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.238214 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dba35d-b975d4787-wmgkr"] Sep 17 18:05:38.260559 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.260537 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph"] Sep 17 18:05:38.262189 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.262171 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dba35d-8594dd6544-cg4ph"] Sep 17 18:05:38.284797 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.284770 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn"] Sep 17 18:05:38.302980 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:38.302945 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dba35d-5b6c5f7c54-r42hn"] Sep 17 18:05:40.151007 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:40.150972 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1961a2-2fe0-4d81-a355-6d7610950e29" path="/var/lib/kubelet/pods/1a1961a2-2fe0-4d81-a355-6d7610950e29/volumes" Sep 17 18:05:40.151548 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:40.151530 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="529e660a-76a9-40a8-b464-fc5bde5e6040" path="/var/lib/kubelet/pods/529e660a-76a9-40a8-b464-fc5bde5e6040/volumes" Sep 17 18:05:40.151901 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:40.151887 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ae4724-ae42-4baa-9a16-e16b664d0597" path="/var/lib/kubelet/pods/72ae4724-ae42-4baa-9a16-e16b664d0597/volumes" Sep 17 18:05:43.496181 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:43.496143 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms"] Sep 17 18:05:58.515313 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.515244 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" podUID="028ba429-364e-4b8d-85f2-df848cc29b26" containerName="main" containerID="cri-o://364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768" gracePeriod=45 Sep 17 18:05:58.753170 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.753148 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:05:58.822414 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822386 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-tmp-dir\") pod \"028ba429-364e-4b8d-85f2-df848cc29b26\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " Sep 17 18:05:58.822414 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822417 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp9fq\" (UniqueName: \"kubernetes.io/projected/028ba429-364e-4b8d-85f2-df848cc29b26-kube-api-access-xp9fq\") pod \"028ba429-364e-4b8d-85f2-df848cc29b26\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " Sep 17 18:05:58.822639 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822439 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028ba429-364e-4b8d-85f2-df848cc29b26-tls-certs\") pod \"028ba429-364e-4b8d-85f2-df848cc29b26\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " Sep 17 18:05:58.822639 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822480 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-model-cache\") pod \"028ba429-364e-4b8d-85f2-df848cc29b26\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " Sep 17 18:05:58.822751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822634 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-home\") pod \"028ba429-364e-4b8d-85f2-df848cc29b26\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " Sep 17 18:05:58.822751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822661 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "028ba429-364e-4b8d-85f2-df848cc29b26" (UID: "028ba429-364e-4b8d-85f2-df848cc29b26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:05:58.822751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822712 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-model-cache" (OuterVolumeSpecName: "model-cache") pod "028ba429-364e-4b8d-85f2-df848cc29b26" (UID: "028ba429-364e-4b8d-85f2-df848cc29b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:05:58.822751 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822740 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-dshm\") pod \"028ba429-364e-4b8d-85f2-df848cc29b26\" (UID: \"028ba429-364e-4b8d-85f2-df848cc29b26\") " Sep 17 18:05:58.822944 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.822847 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-home" (OuterVolumeSpecName: "home") pod "028ba429-364e-4b8d-85f2-df848cc29b26" (UID: "028ba429-364e-4b8d-85f2-df848cc29b26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:05:58.823121 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.823104 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:58.823202 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.823125 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:58.823202 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.823137 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:58.824776 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.824753 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028ba429-364e-4b8d-85f2-df848cc29b26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "028ba429-364e-4b8d-85f2-df848cc29b26" (UID: "028ba429-364e-4b8d-85f2-df848cc29b26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:58.825017 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.824992 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-dshm" (OuterVolumeSpecName: "dshm") pod "028ba429-364e-4b8d-85f2-df848cc29b26" (UID: "028ba429-364e-4b8d-85f2-df848cc29b26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:05:58.825017 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.825003 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028ba429-364e-4b8d-85f2-df848cc29b26-kube-api-access-xp9fq" (OuterVolumeSpecName: "kube-api-access-xp9fq") pod "028ba429-364e-4b8d-85f2-df848cc29b26" (UID: "028ba429-364e-4b8d-85f2-df848cc29b26"). InnerVolumeSpecName "kube-api-access-xp9fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:58.923876 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.923840 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028ba429-364e-4b8d-85f2-df848cc29b26-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:58.923876 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.923868 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp9fq\" (UniqueName: \"kubernetes.io/projected/028ba429-364e-4b8d-85f2-df848cc29b26-kube-api-access-xp9fq\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:58.923876 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:58.923878 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028ba429-364e-4b8d-85f2-df848cc29b26-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:05:59.306439 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.306410 2566 generic.go:356] "Generic (PLEG): container finished" podID="028ba429-364e-4b8d-85f2-df848cc29b26" containerID="364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768" exitCode=0 Sep 17 18:05:59.306606 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.306452 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" Sep 17 18:05:59.306606 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.306493 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" event={"ID":"028ba429-364e-4b8d-85f2-df848cc29b26","Type":"ContainerDied","Data":"364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768"} Sep 17 18:05:59.306606 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.306538 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms" event={"ID":"028ba429-364e-4b8d-85f2-df848cc29b26","Type":"ContainerDied","Data":"a78e1e00a985095f2feffc4e64a315baca51eb52f746bab8b871434a3261607d"} Sep 17 18:05:59.306606 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.306554 2566 scope.go:117] "RemoveContainer" containerID="364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768" Sep 17 18:05:59.315405 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.315387 2566 scope.go:117] "RemoveContainer" containerID="364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768" Sep 17 18:05:59.315706 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:05:59.315684 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768\": container with ID starting with 364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768 not found: ID does not exist" containerID="364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768" Sep 17 18:05:59.315774 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.315717 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768"} err="failed to get container status \"364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768\": rpc error: code = NotFound desc = could not find container \"364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768\": container with ID starting with 364ff0cf2de207e0e21aa0bcb000f37fdea0bcc3bc6620e073cb9a80922f8768 not found: ID does not exist" Sep 17 18:05:59.340131 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.338060 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms"] Sep 17 18:05:59.345640 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:05:59.345620 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-dba35d/ipp-route-dba35d-kserve-6969f9dd8c-bnlms"] Sep 17 18:06:00.150439 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:00.150403 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="028ba429-364e-4b8d-85f2-df848cc29b26" path="/var/lib/kubelet/pods/028ba429-364e-4b8d-85f2-df848cc29b26/volumes" Sep 17 18:06:40.339133 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.339096 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4"] Sep 17 18:06:40.339625 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.339350 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" podUID="92e315cb-5d40-4bb4-9932-ea1d52086cdf" containerName="payload-pre-processing" containerID="cri-o://08c8feed8b3c3c3bfca47a367b52a911f9f5385f7f55f9ce299315cb3744a5d8" gracePeriod=30 Sep 17 18:06:40.509033 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.509002 2566 generic.go:356] "Generic (PLEG): container finished" podID="92e315cb-5d40-4bb4-9932-ea1d52086cdf" containerID="08c8feed8b3c3c3bfca47a367b52a911f9f5385f7f55f9ce299315cb3744a5d8" exitCode=0 Sep 17 18:06:40.509186 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.509041 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" event={"ID":"92e315cb-5d40-4bb4-9932-ea1d52086cdf","Type":"ContainerDied","Data":"08c8feed8b3c3c3bfca47a367b52a911f9f5385f7f55f9ce299315cb3744a5d8"} Sep 17 18:06:40.596897 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.596871 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:06:40.665267 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.665236 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e315cb-5d40-4bb4-9932-ea1d52086cdf-plugins-config-volume\") pod \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " Sep 17 18:06:40.665415 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.665275 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zzgm\" (UniqueName: \"kubernetes.io/projected/92e315cb-5d40-4bb4-9932-ea1d52086cdf-kube-api-access-9zzgm\") pod \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\" (UID: \"92e315cb-5d40-4bb4-9932-ea1d52086cdf\") " Sep 17 18:06:40.665571 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.665550 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e315cb-5d40-4bb4-9932-ea1d52086cdf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92e315cb-5d40-4bb4-9932-ea1d52086cdf" (UID: "92e315cb-5d40-4bb4-9932-ea1d52086cdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:40.667921 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.667888 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e315cb-5d40-4bb4-9932-ea1d52086cdf-kube-api-access-9zzgm" (OuterVolumeSpecName: "kube-api-access-9zzgm") pod "92e315cb-5d40-4bb4-9932-ea1d52086cdf" (UID: "92e315cb-5d40-4bb4-9932-ea1d52086cdf"). InnerVolumeSpecName "kube-api-access-9zzgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:06:40.766631 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.766548 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e315cb-5d40-4bb4-9932-ea1d52086cdf-plugins-config-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:06:40.766631 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:40.766581 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zzgm\" (UniqueName: \"kubernetes.io/projected/92e315cb-5d40-4bb4-9932-ea1d52086cdf-kube-api-access-9zzgm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:06:41.515392 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:41.515358 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" Sep 17 18:06:41.515392 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:41.515381 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4" event={"ID":"92e315cb-5d40-4bb4-9932-ea1d52086cdf","Type":"ContainerDied","Data":"c64a5fba6861e82a468743a09db21b7c6169a58f1ec15927ba7f2ba07e9ae606"} Sep 17 18:06:41.515986 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:41.515450 2566 scope.go:117] "RemoveContainer" containerID="08c8feed8b3c3c3bfca47a367b52a911f9f5385f7f55f9ce299315cb3744a5d8" Sep 17 18:06:41.552653 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:41.552613 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4"] Sep 17 18:06:41.555149 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:41.555129 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b77ab2-8469548bf8-rggs4"] Sep 17 18:06:42.150265 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:42.150230 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92e315cb-5d40-4bb4-9932-ea1d52086cdf" path="/var/lib/kubelet/pods/92e315cb-5d40-4bb4-9932-ea1d52086cdf/volumes" Sep 17 18:06:59.843601 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:59.843561 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn"] Sep 17 18:06:59.844192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:06:59.843821 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" podUID="70e62bff-be8d-4718-ad4f-f32051fb281f" containerName="maas-api" containerID="cri-o://929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b" gracePeriod=30 Sep 17 18:07:00.105036 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.104970 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 18:07:00.233812 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.233779 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-maas-api-tls\") pod \"70e62bff-be8d-4718-ad4f-f32051fb281f\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " Sep 17 18:07:00.234005 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.233821 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4lgj\" (UniqueName: \"kubernetes.io/projected/70e62bff-be8d-4718-ad4f-f32051fb281f-kube-api-access-p4lgj\") pod \"70e62bff-be8d-4718-ad4f-f32051fb281f\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " Sep 17 18:07:00.234005 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.233862 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e62bff-be8d-4718-ad4f-f32051fb281f-openshift-service-ca\") pod \"70e62bff-be8d-4718-ad4f-f32051fb281f\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " Sep 17 18:07:00.234005 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.233937 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-metrics-certs\") pod \"70e62bff-be8d-4718-ad4f-f32051fb281f\" (UID: \"70e62bff-be8d-4718-ad4f-f32051fb281f\") " Sep 17 18:07:00.234276 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.234243 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e62bff-be8d-4718-ad4f-f32051fb281f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70e62bff-be8d-4718-ad4f-f32051fb281f" (UID: "70e62bff-be8d-4718-ad4f-f32051fb281f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:07:00.236162 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.236131 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70e62bff-be8d-4718-ad4f-f32051fb281f" (UID: "70e62bff-be8d-4718-ad4f-f32051fb281f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:07:00.236162 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.236146 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70e62bff-be8d-4718-ad4f-f32051fb281f" (UID: "70e62bff-be8d-4718-ad4f-f32051fb281f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:07:00.236294 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.236173 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e62bff-be8d-4718-ad4f-f32051fb281f-kube-api-access-p4lgj" (OuterVolumeSpecName: "kube-api-access-p4lgj") pod "70e62bff-be8d-4718-ad4f-f32051fb281f" (UID: "70e62bff-be8d-4718-ad4f-f32051fb281f"). InnerVolumeSpecName "kube-api-access-p4lgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:07:00.335297 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.335257 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:00.335297 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.335293 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4lgj\" (UniqueName: \"kubernetes.io/projected/70e62bff-be8d-4718-ad4f-f32051fb281f-kube-api-access-p4lgj\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:00.335545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.335308 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e62bff-be8d-4718-ad4f-f32051fb281f-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:00.335545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.335321 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e62bff-be8d-4718-ad4f-f32051fb281f-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:00.607441 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.607407 2566 generic.go:356] "Generic (PLEG): container finished" podID="70e62bff-be8d-4718-ad4f-f32051fb281f" containerID="929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b" exitCode=0 Sep 17 18:07:00.607629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.607469 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" Sep 17 18:07:00.607629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.607471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" event={"ID":"70e62bff-be8d-4718-ad4f-f32051fb281f","Type":"ContainerDied","Data":"929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b"} Sep 17 18:07:00.607629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.607601 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn" event={"ID":"70e62bff-be8d-4718-ad4f-f32051fb281f","Type":"ContainerDied","Data":"88b7de7ebcf842ee11ea9f623d669b4247278f372db30d63bfcea8e6f30b2759"} Sep 17 18:07:00.607629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.607628 2566 scope.go:117] "RemoveContainer" containerID="929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b" Sep 17 18:07:00.617561 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.617538 2566 scope.go:117] "RemoveContainer" containerID="929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b" Sep 17 18:07:00.617845 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:07:00.617828 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b\": container with ID starting with 929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b not found: ID does not exist" containerID="929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b" Sep 17 18:07:00.617919 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.617852 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b"} err="failed to get container status \"929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b\": rpc error: code = NotFound desc = could not find container \"929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b\": container with ID starting with 929cd7b4059615746404988ff13b266fee0abd2f69fbf2fd3f43206f0e69fd5b not found: ID does not exist" Sep 17 18:07:00.634866 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.634841 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn"] Sep 17 18:07:00.644088 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:00.644065 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-abb363-5bfcd75cf7-4tcgn"] Sep 17 18:07:02.149365 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:02.149320 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e62bff-be8d-4718-ad4f-f32051fb281f" path="/var/lib/kubelet/pods/70e62bff-be8d-4718-ad4f-f32051fb281f/volumes" Sep 17 18:07:33.526587 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.526549 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b777b4495-bq469"] Sep 17 18:07:33.527133 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.526808 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b777b4495-bq469" podUID="73c3b99c-fdac-4a97-b23a-39b928794969" containerName="manager" containerID="cri-o://0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02" gracePeriod=30 Sep 17 18:07:33.778028 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.777944 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 18:07:33.781252 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.781224 2566 generic.go:356] "Generic (PLEG): container finished" podID="73c3b99c-fdac-4a97-b23a-39b928794969" containerID="0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02" exitCode=0 Sep 17 18:07:33.781381 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.781267 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-bq469" Sep 17 18:07:33.781381 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.781304 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-bq469" event={"ID":"73c3b99c-fdac-4a97-b23a-39b928794969","Type":"ContainerDied","Data":"0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02"} Sep 17 18:07:33.781381 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.781373 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-bq469" event={"ID":"73c3b99c-fdac-4a97-b23a-39b928794969","Type":"ContainerDied","Data":"5e25e61883436b70fc13963e38b6da3c901f8e1c772004303ad59cac74d59d3a"} Sep 17 18:07:33.781497 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.781403 2566 scope.go:117] "RemoveContainer" containerID="0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02" Sep 17 18:07:33.790196 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.790180 2566 scope.go:117] "RemoveContainer" containerID="0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02" Sep 17 18:07:33.790438 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:07:33.790420 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02\": container with ID starting with 0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02 not found: ID does not exist" containerID="0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02" Sep 17 18:07:33.790489 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.790449 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02"} err="failed to get container status \"0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02\": rpc error: code = NotFound desc = could not find container \"0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02\": container with ID starting with 0b86d8803b10bc482b36ff59f9deee3479356198a3452ec686c4400c05b8ee02 not found: ID does not exist" Sep 17 18:07:33.910744 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.910712 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/73c3b99c-fdac-4a97-b23a-39b928794969-kube-api-access-9xmtd\") pod \"73c3b99c-fdac-4a97-b23a-39b928794969\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " Sep 17 18:07:33.910907 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.910756 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert\") pod \"73c3b99c-fdac-4a97-b23a-39b928794969\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " Sep 17 18:07:33.910907 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.910789 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-metrics-certs\") pod \"73c3b99c-fdac-4a97-b23a-39b928794969\" (UID: \"73c3b99c-fdac-4a97-b23a-39b928794969\") " Sep 17 18:07:33.912907 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.912872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "73c3b99c-fdac-4a97-b23a-39b928794969" (UID: "73c3b99c-fdac-4a97-b23a-39b928794969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:07:33.913043 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.912916 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "73c3b99c-fdac-4a97-b23a-39b928794969" (UID: "73c3b99c-fdac-4a97-b23a-39b928794969"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:07:33.913043 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:33.912972 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c3b99c-fdac-4a97-b23a-39b928794969-kube-api-access-9xmtd" (OuterVolumeSpecName: "kube-api-access-9xmtd") pod "73c3b99c-fdac-4a97-b23a-39b928794969" (UID: "73c3b99c-fdac-4a97-b23a-39b928794969"). InnerVolumeSpecName "kube-api-access-9xmtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:07:34.011572 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:34.011541 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:34.011572 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:34.011567 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/73c3b99c-fdac-4a97-b23a-39b928794969-kube-api-access-9xmtd\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:34.011572 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:34.011577 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c3b99c-fdac-4a97-b23a-39b928794969-webhook-cert\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:07:34.110339 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:34.110312 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b777b4495-bq469"] Sep 17 18:07:34.121723 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:34.121697 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b777b4495-bq469"] Sep 17 18:07:34.149367 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:07:34.149342 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c3b99c-fdac-4a97-b23a-39b928794969" path="/var/lib/kubelet/pods/73c3b99c-fdac-4a97-b23a-39b928794969/volumes" Sep 17 18:08:34.976724 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.976689 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b777b4495-vdwlf"] Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977173 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529e660a-76a9-40a8-b464-fc5bde5e6040" containerName="payload-processing" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977192 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="529e660a-76a9-40a8-b464-fc5bde5e6040" containerName="payload-processing" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977206 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1961a2-2fe0-4d81-a355-6d7610950e29" containerName="maas-api" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977215 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1961a2-2fe0-4d81-a355-6d7610950e29" containerName="maas-api" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977226 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e62bff-be8d-4718-ad4f-f32051fb281f" containerName="maas-api" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977235 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e62bff-be8d-4718-ad4f-f32051fb281f" containerName="maas-api" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977248 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c3b99c-fdac-4a97-b23a-39b928794969" containerName="manager" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977258 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c3b99c-fdac-4a97-b23a-39b928794969" containerName="manager" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977278 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ae4724-ae42-4baa-9a16-e16b664d0597" containerName="payload-pre-processing" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977287 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ae4724-ae42-4baa-9a16-e16b664d0597" containerName="payload-pre-processing" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977295 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="028ba429-364e-4b8d-85f2-df848cc29b26" containerName="main" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977303 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="028ba429-364e-4b8d-85f2-df848cc29b26" containerName="main" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977319 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e315cb-5d40-4bb4-9932-ea1d52086cdf" containerName="payload-pre-processing" Sep 17 18:08:34.977325 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977327 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e315cb-5d40-4bb4-9932-ea1d52086cdf" containerName="payload-pre-processing" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977341 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a397e13-5928-4850-868d-6c93ab8a3b3a" containerName="payload-pre-processing" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977349 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a397e13-5928-4850-868d-6c93ab8a3b3a" containerName="payload-pre-processing" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977427 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a397e13-5928-4850-868d-6c93ab8a3b3a" containerName="payload-pre-processing" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977438 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c3b99c-fdac-4a97-b23a-39b928794969" containerName="manager" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977448 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ae4724-ae42-4baa-9a16-e16b664d0597" containerName="payload-pre-processing" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977458 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e62bff-be8d-4718-ad4f-f32051fb281f" containerName="maas-api" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977470 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e315cb-5d40-4bb4-9932-ea1d52086cdf" containerName="payload-pre-processing" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977480 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="028ba429-364e-4b8d-85f2-df848cc29b26" containerName="main" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977491 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1961a2-2fe0-4d81-a355-6d7610950e29" containerName="maas-api" Sep 17 18:08:34.978042 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.977500 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="529e660a-76a9-40a8-b464-fc5bde5e6040" containerName="payload-processing" Sep 17 18:08:34.980623 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.980604 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:34.984495 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.984474 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 18:08:34.985320 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.985302 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 18:08:34.985385 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.985311 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 18:08:34.985425 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:34.985397 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tcnq8\"" Sep 17 18:08:35.002565 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.002540 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b777b4495-vdwlf"] Sep 17 18:08:35.091664 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.091626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4px4\" (UniqueName: \"kubernetes.io/projected/abae5133-75d8-48b6-bdbe-b46a2efe4616-kube-api-access-x4px4\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.091836 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.091716 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-metrics-certs\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.091836 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.091755 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-webhook-cert\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.192469 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.192443 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-webhook-cert\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.192619 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.192481 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4px4\" (UniqueName: \"kubernetes.io/projected/abae5133-75d8-48b6-bdbe-b46a2efe4616-kube-api-access-x4px4\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.192619 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.192525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-metrics-certs\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.194889 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.194858 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-webhook-cert\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.194889 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.194882 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-metrics-certs\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.208064 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.208046 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4px4\" (UniqueName: \"kubernetes.io/projected/abae5133-75d8-48b6-bdbe-b46a2efe4616-kube-api-access-x4px4\") pod \"maas-controller-6b777b4495-vdwlf\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.290669 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.290605 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:35.576909 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.576886 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b777b4495-vdwlf"] Sep 17 18:08:35.578895 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:08:35.578872 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabae5133_75d8_48b6_bdbe_b46a2efe4616.slice/crio-493d73082d58b2f257b843a24980134c010a74231a94000a9f757513dbac0995 WatchSource:0}: Error finding container 493d73082d58b2f257b843a24980134c010a74231a94000a9f757513dbac0995: Status 404 returned error can't find the container with id 493d73082d58b2f257b843a24980134c010a74231a94000a9f757513dbac0995 Sep 17 18:08:35.580200 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:35.580181 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:08:36.057649 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:36.057613 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-vdwlf" event={"ID":"abae5133-75d8-48b6-bdbe-b46a2efe4616","Type":"ContainerStarted","Data":"493d73082d58b2f257b843a24980134c010a74231a94000a9f757513dbac0995"} Sep 17 18:08:37.063024 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:37.062987 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-vdwlf" event={"ID":"abae5133-75d8-48b6-bdbe-b46a2efe4616","Type":"ContainerStarted","Data":"430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3"} Sep 17 18:08:37.063454 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:37.063178 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:48.069459 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:48.069424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:48.092435 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:48.092389 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b777b4495-vdwlf" podStartSLOduration=13.532630793 podStartE2EDuration="14.092375843s" totalImagesPullingTime="559.74505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:37.088277885 +0000 UTC m=+2253.650442756" watchObservedRunningTime="2026-09-17 18:08:48.092375843 +0000 UTC m=+2264.654540719" Sep 17 18:08:57.418068 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.418027 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b777b4495-vdwlf"] Sep 17 18:08:57.418558 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.418326 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b777b4495-vdwlf" podUID="abae5133-75d8-48b6-bdbe-b46a2efe4616" containerName="manager" containerID="cri-o://430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3" gracePeriod=30 Sep 17 18:08:57.653693 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.653671 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:57.679440 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.679381 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-metrics-certs\") pod \"abae5133-75d8-48b6-bdbe-b46a2efe4616\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " Sep 17 18:08:57.679550 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.679460 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-webhook-cert\") pod \"abae5133-75d8-48b6-bdbe-b46a2efe4616\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " Sep 17 18:08:57.679550 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.679499 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4px4\" (UniqueName: \"kubernetes.io/projected/abae5133-75d8-48b6-bdbe-b46a2efe4616-kube-api-access-x4px4\") pod \"abae5133-75d8-48b6-bdbe-b46a2efe4616\" (UID: \"abae5133-75d8-48b6-bdbe-b46a2efe4616\") " Sep 17 18:08:57.681555 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.681513 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "abae5133-75d8-48b6-bdbe-b46a2efe4616" (UID: "abae5133-75d8-48b6-bdbe-b46a2efe4616"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:08:57.681661 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.681563 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "abae5133-75d8-48b6-bdbe-b46a2efe4616" (UID: "abae5133-75d8-48b6-bdbe-b46a2efe4616"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:08:57.681806 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.681782 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abae5133-75d8-48b6-bdbe-b46a2efe4616-kube-api-access-x4px4" (OuterVolumeSpecName: "kube-api-access-x4px4") pod "abae5133-75d8-48b6-bdbe-b46a2efe4616" (UID: "abae5133-75d8-48b6-bdbe-b46a2efe4616"). InnerVolumeSpecName "kube-api-access-x4px4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:08:57.780192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.780154 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:08:57.780192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.780187 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abae5133-75d8-48b6-bdbe-b46a2efe4616-webhook-cert\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:08:57.780192 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:57.780196 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4px4\" (UniqueName: \"kubernetes.io/projected/abae5133-75d8-48b6-bdbe-b46a2efe4616-kube-api-access-x4px4\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:08:58.157974 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.157918 2566 generic.go:356] "Generic (PLEG): container finished" podID="abae5133-75d8-48b6-bdbe-b46a2efe4616" containerID="430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3" exitCode=0 Sep 17 18:08:58.158115 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.157995 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-vdwlf" Sep 17 18:08:58.158115 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.157997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-vdwlf" event={"ID":"abae5133-75d8-48b6-bdbe-b46a2efe4616","Type":"ContainerDied","Data":"430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3"} Sep 17 18:08:58.158115 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.158048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-vdwlf" event={"ID":"abae5133-75d8-48b6-bdbe-b46a2efe4616","Type":"ContainerDied","Data":"493d73082d58b2f257b843a24980134c010a74231a94000a9f757513dbac0995"} Sep 17 18:08:58.158115 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.158070 2566 scope.go:117] "RemoveContainer" containerID="430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3" Sep 17 18:08:58.166938 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.166917 2566 scope.go:117] "RemoveContainer" containerID="430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3" Sep 17 18:08:58.167301 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:08:58.167277 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3\": container with ID starting with 430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3 not found: ID does not exist" containerID="430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3" Sep 17 18:08:58.167370 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.167323 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3"} err="failed to get container status \"430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3\": rpc error: code = NotFound desc = could not find container \"430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3\": container with ID starting with 430e9aa884b0f7378fcc95fd43044f9b8730bad099024ddf0af838f5303412b3 not found: ID does not exist" Sep 17 18:08:58.180012 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.179980 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b777b4495-vdwlf"] Sep 17 18:08:58.186336 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:58.186314 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b777b4495-vdwlf"] Sep 17 18:08:59.004733 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.004697 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b777b4495-8mxgh"] Sep 17 18:08:59.005136 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.005028 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abae5133-75d8-48b6-bdbe-b46a2efe4616" containerName="manager" Sep 17 18:08:59.005136 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.005039 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abae5133-75d8-48b6-bdbe-b46a2efe4616" containerName="manager" Sep 17 18:08:59.005136 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.005100 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abae5133-75d8-48b6-bdbe-b46a2efe4616" containerName="manager" Sep 17 18:08:59.019581 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.019558 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.026009 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.025936 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 18:08:59.026092 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.026062 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 18:08:59.026092 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.025936 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tcnq8\"" Sep 17 18:08:59.026206 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.026171 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 18:08:59.042218 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.042196 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b777b4495-8mxgh"] Sep 17 18:08:59.091813 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.091786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5471202-2060-45dd-8fa4-c330b636b73d-webhook-cert\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.091925 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.091817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5471202-2060-45dd-8fa4-c330b636b73d-metrics-certs\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.091925 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.091855 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgfx\" (UniqueName: \"kubernetes.io/projected/d5471202-2060-45dd-8fa4-c330b636b73d-kube-api-access-kdgfx\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.192199 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.192171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5471202-2060-45dd-8fa4-c330b636b73d-webhook-cert\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.192199 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.192200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5471202-2060-45dd-8fa4-c330b636b73d-metrics-certs\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.192388 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.192257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgfx\" (UniqueName: \"kubernetes.io/projected/d5471202-2060-45dd-8fa4-c330b636b73d-kube-api-access-kdgfx\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.194675 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.194647 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5471202-2060-45dd-8fa4-c330b636b73d-webhook-cert\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.194779 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.194681 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5471202-2060-45dd-8fa4-c330b636b73d-metrics-certs\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.202252 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.202230 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgfx\" (UniqueName: \"kubernetes.io/projected/d5471202-2060-45dd-8fa4-c330b636b73d-kube-api-access-kdgfx\") pod \"maas-controller-6b777b4495-8mxgh\" (UID: \"d5471202-2060-45dd-8fa4-c330b636b73d\") " pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.329424 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.329393 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:08:59.612080 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:08:59.612048 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b777b4495-8mxgh"] Sep 17 18:08:59.612794 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:08:59.612770 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5471202_2060_45dd_8fa4_c330b636b73d.slice/crio-b1b1447de3abbd8956099985b1ab3b0ea567a988f463ff458345bcff5a62ca5c WatchSource:0}: Error finding container b1b1447de3abbd8956099985b1ab3b0ea567a988f463ff458345bcff5a62ca5c: Status 404 returned error can't find the container with id b1b1447de3abbd8956099985b1ab3b0ea567a988f463ff458345bcff5a62ca5c Sep 17 18:09:00.150678 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:09:00.150644 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abae5133-75d8-48b6-bdbe-b46a2efe4616" path="/var/lib/kubelet/pods/abae5133-75d8-48b6-bdbe-b46a2efe4616/volumes" Sep 17 18:09:00.169682 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:09:00.169647 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-8mxgh" event={"ID":"d5471202-2060-45dd-8fa4-c330b636b73d","Type":"ContainerStarted","Data":"f6537ad5b2ffb22ea2296edaa3e74ce078b2d9a253375f95105425e2dc41a05e"} Sep 17 18:09:00.169825 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:09:00.169700 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b777b4495-8mxgh" event={"ID":"d5471202-2060-45dd-8fa4-c330b636b73d","Type":"ContainerStarted","Data":"b1b1447de3abbd8956099985b1ab3b0ea567a988f463ff458345bcff5a62ca5c"} Sep 17 18:09:00.169825 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:09:00.169786 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:09:00.203008 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:09:00.202904 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b777b4495-8mxgh" podStartSLOduration=1.865892425 podStartE2EDuration="2.202890426s" totalImagesPullingTime="336.998001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:00.19962223 +0000 UTC m=+2276.761787090" watchObservedRunningTime="2026-09-17 18:09:00.202890426 +0000 UTC m=+2276.765055286" Sep 17 18:09:11.177816 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:09:11.177774 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b777b4495-8mxgh" Sep 17 18:10:22.916967 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:22.916872 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69"] Sep 17 18:10:22.917510 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:22.917110 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" containerID="cri-o://7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986" gracePeriod=10 Sep 17 18:10:23.466056 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.466017 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 18:10:23.541345 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.541252 2566 generic.go:356] "Generic (PLEG): container finished" podID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerID="7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986" exitCode=0 Sep 17 18:10:23.541489 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.541329 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" Sep 17 18:10:23.541489 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.541333 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" event={"ID":"5daa7083-d1ec-4069-a7d6-9a3fec0574e6","Type":"ContainerDied","Data":"7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986"} Sep 17 18:10:23.541489 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.541484 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69" event={"ID":"5daa7083-d1ec-4069-a7d6-9a3fec0574e6","Type":"ContainerDied","Data":"813b40557ab6997dfe2541570b182ef1ae70c7a9500914f0fcbf4e9da5105b0f"} Sep 17 18:10:23.541617 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.541511 2566 scope.go:117] "RemoveContainer" containerID="7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986" Sep 17 18:10:23.552854 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.552832 2566 scope.go:117] "RemoveContainer" containerID="a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e" Sep 17 18:10:23.560876 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.560856 2566 scope.go:117] "RemoveContainer" containerID="7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986" Sep 17 18:10:23.561193 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:10:23.561169 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986\": container with ID starting with 7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986 not found: ID does not exist" containerID="7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986" Sep 17 18:10:23.561302 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.561200 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986"} err="failed to get container status \"7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986\": rpc error: code = NotFound desc = could not find container \"7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986\": container with ID starting with 7d70ae5b3bac611e5d213c08f85d74013d307c81bea0b9b433f8b2e2798eb986 not found: ID does not exist" Sep 17 18:10:23.561302 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.561219 2566 scope.go:117] "RemoveContainer" containerID="a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e" Sep 17 18:10:23.561496 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:10:23.561476 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e\": container with ID starting with a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e not found: ID does not exist" containerID="a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e" Sep 17 18:10:23.561558 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.561506 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e"} err="failed to get container status \"a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e\": rpc error: code = NotFound desc = could not find container \"a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e\": container with ID starting with a025711f0c98ef660eca9e38ede654cefcf50597ee7065ca75c81e31f5931d7e not found: ID does not exist" Sep 17 18:10:23.605901 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.605856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-extensions-socket-volume\") pod \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " Sep 17 18:10:23.606091 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.605916 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59gdf\" (UniqueName: \"kubernetes.io/projected/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-kube-api-access-59gdf\") pod \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\" (UID: \"5daa7083-d1ec-4069-a7d6-9a3fec0574e6\") " Sep 17 18:10:23.606300 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.606272 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5daa7083-d1ec-4069-a7d6-9a3fec0574e6" (UID: "5daa7083-d1ec-4069-a7d6-9a3fec0574e6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:10:23.607947 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.607923 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-kube-api-access-59gdf" (OuterVolumeSpecName: "kube-api-access-59gdf") pod "5daa7083-d1ec-4069-a7d6-9a3fec0574e6" (UID: "5daa7083-d1ec-4069-a7d6-9a3fec0574e6"). InnerVolumeSpecName "kube-api-access-59gdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:10:23.707191 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.707158 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-extensions-socket-volume\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:10:23.707191 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.707186 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59gdf\" (UniqueName: \"kubernetes.io/projected/5daa7083-d1ec-4069-a7d6-9a3fec0574e6-kube-api-access-59gdf\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:10:23.865055 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.865024 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69"] Sep 17 18:10:23.870070 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:23.870045 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5tw69"] Sep 17 18:10:24.150126 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:10:24.150050 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" path="/var/lib/kubelet/pods/5daa7083-d1ec-4069-a7d6-9a3fec0574e6/volumes" Sep 17 18:11:23.600080 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600043 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp"] Sep 17 18:11:23.600531 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600389 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" Sep 17 18:11:23.600531 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600401 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" Sep 17 18:11:23.600531 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600414 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" Sep 17 18:11:23.600531 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600419 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" Sep 17 18:11:23.600531 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600491 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" Sep 17 18:11:23.600531 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.600500 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daa7083-d1ec-4069-a7d6-9a3fec0574e6" containerName="manager" Sep 17 18:11:23.603406 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.603389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.607246 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.607230 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wwhmw\"" Sep 17 18:11:23.620102 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.620072 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp"] Sep 17 18:11:23.725457 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.725422 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6308ccf3-d7f0-4e23-9de7-8f3c52251a62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v6czp\" (UID: \"6308ccf3-d7f0-4e23-9de7-8f3c52251a62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.725630 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.725479 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cf8g\" (UniqueName: \"kubernetes.io/projected/6308ccf3-d7f0-4e23-9de7-8f3c52251a62-kube-api-access-6cf8g\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v6czp\" (UID: \"6308ccf3-d7f0-4e23-9de7-8f3c52251a62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.826157 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.826129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6308ccf3-d7f0-4e23-9de7-8f3c52251a62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v6czp\" (UID: \"6308ccf3-d7f0-4e23-9de7-8f3c52251a62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.826358 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.826177 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cf8g\" (UniqueName: \"kubernetes.io/projected/6308ccf3-d7f0-4e23-9de7-8f3c52251a62-kube-api-access-6cf8g\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v6czp\" (UID: \"6308ccf3-d7f0-4e23-9de7-8f3c52251a62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.826545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.826522 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6308ccf3-d7f0-4e23-9de7-8f3c52251a62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v6czp\" (UID: \"6308ccf3-d7f0-4e23-9de7-8f3c52251a62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.835463 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.835440 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cf8g\" (UniqueName: \"kubernetes.io/projected/6308ccf3-d7f0-4e23-9de7-8f3c52251a62-kube-api-access-6cf8g\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v6czp\" (UID: \"6308ccf3-d7f0-4e23-9de7-8f3c52251a62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:23.913637 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:23.913559 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:24.188818 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:24.188792 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp"] Sep 17 18:11:24.190326 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:11:24.190295 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6308ccf3_d7f0_4e23_9de7_8f3c52251a62.slice/crio-f0c42988d7dd226a7544af049052b68b69b6871a15219f3732651be120e493b3 WatchSource:0}: Error finding container f0c42988d7dd226a7544af049052b68b69b6871a15219f3732651be120e493b3: Status 404 returned error can't find the container with id f0c42988d7dd226a7544af049052b68b69b6871a15219f3732651be120e493b3 Sep 17 18:11:24.818304 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:24.818270 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" event={"ID":"6308ccf3-d7f0-4e23-9de7-8f3c52251a62","Type":"ContainerStarted","Data":"b0e4a327f0dbb2a6774d1598b20408fc63a1e09f8ca702230072f1a668814105"} Sep 17 18:11:24.818732 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:24.818320 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" event={"ID":"6308ccf3-d7f0-4e23-9de7-8f3c52251a62","Type":"ContainerStarted","Data":"f0c42988d7dd226a7544af049052b68b69b6871a15219f3732651be120e493b3"} Sep 17 18:11:24.818732 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:24.818377 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:24.848577 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:24.848523 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" podStartSLOduration=1.848508747 podStartE2EDuration="1.848508747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:24.847498468 +0000 UTC m=+2421.409663310" watchObservedRunningTime="2026-09-17 18:11:24.848508747 +0000 UTC m=+2421.410673607" Sep 17 18:11:35.825375 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:35.825338 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v6czp" Sep 17 18:11:58.926626 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:58.926591 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr"] Sep 17 18:11:58.930077 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:58.930059 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:58.932592 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:58.932575 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c1efc029-kserve-self-signed-certs\"" Sep 17 18:11:58.938317 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:58.938296 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr"] Sep 17 18:11:59.037122 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.037089 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-home\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.037361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.037132 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tmp-dir\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.037361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.037178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-model-cache\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.037361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.037224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-dshm\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.037361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.037261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26q7w\" (UniqueName: \"kubernetes.io/projected/70f7ba22-5739-47f3-a14d-43b35c8b46ee-kube-api-access-26q7w\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.037361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.037298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tls-certs\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138187 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-home\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138187 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138195 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tmp-dir\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138427 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138211 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-model-cache\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138427 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-dshm\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138427 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26q7w\" (UniqueName: \"kubernetes.io/projected/70f7ba22-5739-47f3-a14d-43b35c8b46ee-kube-api-access-26q7w\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138427 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138331 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tls-certs\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138648 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138626 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-home\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138648 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138640 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tmp-dir\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.138743 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.138689 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-model-cache\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.140642 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.140604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-dshm\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.140823 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.140804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tls-certs\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.147065 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.147045 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26q7w\" (UniqueName: \"kubernetes.io/projected/70f7ba22-5739-47f3-a14d-43b35c8b46ee-kube-api-access-26q7w\") pod \"e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.239896 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.239815 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:11:59.518108 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.517927 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr"] Sep 17 18:11:59.520415 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:11:59.520393 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f7ba22_5739_47f3_a14d_43b35c8b46ee.slice/crio-997236e568a5d91de56f31e732e8393993cd7d2555b55e18a2bcac46ed09b94d WatchSource:0}: Error finding container 997236e568a5d91de56f31e732e8393993cd7d2555b55e18a2bcac46ed09b94d: Status 404 returned error can't find the container with id 997236e568a5d91de56f31e732e8393993cd7d2555b55e18a2bcac46ed09b94d Sep 17 18:11:59.977218 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.977171 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" event={"ID":"70f7ba22-5739-47f3-a14d-43b35c8b46ee","Type":"ContainerStarted","Data":"d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814"} Sep 17 18:11:59.977822 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.977229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" event={"ID":"70f7ba22-5739-47f3-a14d-43b35c8b46ee","Type":"ContainerStarted","Data":"997236e568a5d91de56f31e732e8393993cd7d2555b55e18a2bcac46ed09b94d"} Sep 17 18:11:59.997172 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:11:59.997124 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" podStartSLOduration=1.9971096400000001 podStartE2EDuration="1.99710964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:59.99611064 +0000 UTC m=+2456.558275510" watchObservedRunningTime="2026-09-17 18:11:59.99710964 +0000 UTC m=+2456.559274499" Sep 17 18:12:09.240254 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:09.240221 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:12:09.242976 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:09.240530 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:12:09.252855 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:09.252837 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:12:10.039632 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:10.039598 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:12:19.736538 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.736494 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k"] Sep 17 18:12:19.740150 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.740126 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:19.742621 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.742599 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c1efc029-kserve-self-signed-certs\"" Sep 17 18:12:19.749400 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.749376 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k"] Sep 17 18:12:19.925360 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.925324 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mdv\" (UniqueName: \"kubernetes.io/projected/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-kube-api-access-25mdv\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:19.925550 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.925382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tmp-dir\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:19.925550 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.925497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-dshm\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:19.925550 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.925531 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tls-certs\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:19.925696 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.925593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-model-cache\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:19.925696 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:19.925615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-home\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.026770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.026678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mdv\" (UniqueName: \"kubernetes.io/projected/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-kube-api-access-25mdv\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.026770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.026738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tmp-dir\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027064 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.026786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-dshm\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027064 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.026810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tls-certs\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027064 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.026850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-model-cache\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027064 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.027005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-home\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027283 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.027197 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tmp-dir\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027283 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.027202 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-model-cache\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.027393 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.027357 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-home\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.029227 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.029193 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-dshm\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.029513 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.029491 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tls-certs\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.034853 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.034830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mdv\" (UniqueName: \"kubernetes.io/projected/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-kube-api-access-25mdv\") pod \"e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.050041 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.050020 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:20.323669 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:20.323555 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k"] Sep 17 18:12:20.325997 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:12:20.325969 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f8c37d_87c1_480a_84fa_e4d7c4dae3cb.slice/crio-8efb669d7c008c2068f26a313fc5aaf8ed54a172930ea2ce2544e42899fd3f00 WatchSource:0}: Error finding container 8efb669d7c008c2068f26a313fc5aaf8ed54a172930ea2ce2544e42899fd3f00: Status 404 returned error can't find the container with id 8efb669d7c008c2068f26a313fc5aaf8ed54a172930ea2ce2544e42899fd3f00 Sep 17 18:12:21.082510 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:21.082471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" event={"ID":"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb","Type":"ContainerStarted","Data":"eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7"} Sep 17 18:12:21.082510 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:21.082515 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" event={"ID":"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb","Type":"ContainerStarted","Data":"8efb669d7c008c2068f26a313fc5aaf8ed54a172930ea2ce2544e42899fd3f00"} Sep 17 18:12:21.104376 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:21.104332 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" podStartSLOduration=2.104313801 podStartE2EDuration="2.104313801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:21.103823666 +0000 UTC m=+2477.665988526" watchObservedRunningTime="2026-09-17 18:12:21.104313801 +0000 UTC m=+2477.666478662" Sep 17 18:12:30.050813 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:30.050767 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:30.051315 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:30.050832 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:30.063102 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:30.063076 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:30.136158 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:30.136126 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:33.853730 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:33.853697 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr"] Sep 17 18:12:34.919629 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:34.919597 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k"] Sep 17 18:12:48.874350 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:48.874300 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" podUID="70f7ba22-5739-47f3-a14d-43b35c8b46ee" containerName="main" containerID="cri-o://d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814" gracePeriod=45 Sep 17 18:12:49.130449 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.130396 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:12:49.176125 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176091 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tmp-dir\") pod \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " Sep 17 18:12:49.176307 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176172 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tls-certs\") pod \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " Sep 17 18:12:49.176307 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176215 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-dshm\") pod \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " Sep 17 18:12:49.176307 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176249 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-model-cache\") pod \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " Sep 17 18:12:49.176307 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176272 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26q7w\" (UniqueName: \"kubernetes.io/projected/70f7ba22-5739-47f3-a14d-43b35c8b46ee-kube-api-access-26q7w\") pod \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " Sep 17 18:12:49.176545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176309 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70f7ba22-5739-47f3-a14d-43b35c8b46ee" (UID: "70f7ba22-5739-47f3-a14d-43b35c8b46ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:49.176545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176324 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-home\") pod \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\" (UID: \"70f7ba22-5739-47f3-a14d-43b35c8b46ee\") " Sep 17 18:12:49.176545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176493 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "70f7ba22-5739-47f3-a14d-43b35c8b46ee" (UID: "70f7ba22-5739-47f3-a14d-43b35c8b46ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:49.176545 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176538 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-home" (OuterVolumeSpecName: "home") pod "70f7ba22-5739-47f3-a14d-43b35c8b46ee" (UID: "70f7ba22-5739-47f3-a14d-43b35c8b46ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:49.176749 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176650 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:49.176749 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176668 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:49.176749 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.176677 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:49.178459 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.178439 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-dshm" (OuterVolumeSpecName: "dshm") pod "70f7ba22-5739-47f3-a14d-43b35c8b46ee" (UID: "70f7ba22-5739-47f3-a14d-43b35c8b46ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:49.178586 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.178566 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70f7ba22-5739-47f3-a14d-43b35c8b46ee" (UID: "70f7ba22-5739-47f3-a14d-43b35c8b46ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:12:49.178719 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.178704 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f7ba22-5739-47f3-a14d-43b35c8b46ee-kube-api-access-26q7w" (OuterVolumeSpecName: "kube-api-access-26q7w") pod "70f7ba22-5739-47f3-a14d-43b35c8b46ee" (UID: "70f7ba22-5739-47f3-a14d-43b35c8b46ee"). InnerVolumeSpecName "kube-api-access-26q7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:12:49.221233 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.221205 2566 generic.go:356] "Generic (PLEG): container finished" podID="70f7ba22-5739-47f3-a14d-43b35c8b46ee" containerID="d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814" exitCode=0 Sep 17 18:12:49.221361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.221261 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" Sep 17 18:12:49.221361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.221295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" event={"ID":"70f7ba22-5739-47f3-a14d-43b35c8b46ee","Type":"ContainerDied","Data":"d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814"} Sep 17 18:12:49.221361 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.221341 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr" event={"ID":"70f7ba22-5739-47f3-a14d-43b35c8b46ee","Type":"ContainerDied","Data":"997236e568a5d91de56f31e732e8393993cd7d2555b55e18a2bcac46ed09b94d"} Sep 17 18:12:49.221470 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.221362 2566 scope.go:117] "RemoveContainer" containerID="d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814" Sep 17 18:12:49.229799 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.229782 2566 scope.go:117] "RemoveContainer" containerID="d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814" Sep 17 18:12:49.230066 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:12:49.230051 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814\": container with ID starting with d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814 not found: ID does not exist" containerID="d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814" Sep 17 18:12:49.230116 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.230073 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814"} err="failed to get container status \"d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814\": rpc error: code = NotFound desc = could not find container \"d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814\": container with ID starting with d936cc5a85593ca68d4cd7f2faa530d2ed8eecd3c9b55222742dacbe64bef814 not found: ID does not exist" Sep 17 18:12:49.241232 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.241209 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr"] Sep 17 18:12:49.244826 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.244802 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c1efc029-kserve-5ffc6fcd7-dljwr"] Sep 17 18:12:49.277780 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.277754 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f7ba22-5739-47f3-a14d-43b35c8b46ee-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:49.277780 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.277782 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f7ba22-5739-47f3-a14d-43b35c8b46ee-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:49.278017 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.277797 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26q7w\" (UniqueName: \"kubernetes.io/projected/70f7ba22-5739-47f3-a14d-43b35c8b46ee-kube-api-access-26q7w\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:49.941254 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:49.941216 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" podUID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" containerName="main" containerID="cri-o://eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7" gracePeriod=45 Sep 17 18:12:50.150382 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.150322 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f7ba22-5739-47f3-a14d-43b35c8b46ee" path="/var/lib/kubelet/pods/70f7ba22-5739-47f3-a14d-43b35c8b46ee/volumes" Sep 17 18:12:50.186657 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.186632 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:50.228421 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.228328 2566 generic.go:356] "Generic (PLEG): container finished" podID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" containerID="eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7" exitCode=0 Sep 17 18:12:50.228421 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.228372 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" Sep 17 18:12:50.228653 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.228417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" event={"ID":"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb","Type":"ContainerDied","Data":"eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7"} Sep 17 18:12:50.228653 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.228461 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" event={"ID":"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb","Type":"ContainerDied","Data":"8efb669d7c008c2068f26a313fc5aaf8ed54a172930ea2ce2544e42899fd3f00"} Sep 17 18:12:50.228653 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.228480 2566 scope.go:117] "RemoveContainer" containerID="eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7" Sep 17 18:12:50.236925 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.236906 2566 scope.go:117] "RemoveContainer" containerID="eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7" Sep 17 18:12:50.237239 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:12:50.237219 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7\": container with ID starting with eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7 not found: ID does not exist" containerID="eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7" Sep 17 18:12:50.237306 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.237245 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7"} err="failed to get container status \"eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7\": rpc error: code = NotFound desc = could not find container \"eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7\": container with ID starting with eb514c212f22d928e00efa9ec405840a57c28440066660dee45f2bcf6e5f00c7 not found: ID does not exist" Sep 17 18:12:50.287343 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tmp-dir\") pod \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " Sep 17 18:12:50.287506 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287382 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-dshm\") pod \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " Sep 17 18:12:50.287506 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tls-certs\") pod \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " Sep 17 18:12:50.287506 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287473 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25mdv\" (UniqueName: \"kubernetes.io/projected/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-kube-api-access-25mdv\") pod \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " Sep 17 18:12:50.287506 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287502 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-home\") pod \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " Sep 17 18:12:50.287744 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287540 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-model-cache\") pod \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\" (UID: \"35f8c37d-87c1-480a-84fa-e4d7c4dae3cb\") " Sep 17 18:12:50.287744 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287618 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" (UID: "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:50.287857 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287774 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tmp-dir\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:50.287857 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287783 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-home" (OuterVolumeSpecName: "home") pod "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" (UID: "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:50.287936 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.287855 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" (UID: "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:50.289567 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.289537 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-kube-api-access-25mdv" (OuterVolumeSpecName: "kube-api-access-25mdv") pod "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" (UID: "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb"). InnerVolumeSpecName "kube-api-access-25mdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:12:50.289634 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.289566 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" (UID: "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:12:50.289634 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.289624 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-dshm" (OuterVolumeSpecName: "dshm") pod "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" (UID: "35f8c37d-87c1-480a-84fa-e4d7c4dae3cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:12:50.389117 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.389087 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mdv\" (UniqueName: \"kubernetes.io/projected/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-kube-api-access-25mdv\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:50.389117 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.389114 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-home\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:50.389292 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.389124 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-model-cache\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:50.389292 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.389133 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-dshm\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:50.389292 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.389140 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb-tls-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:12:50.550491 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.550457 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k"] Sep 17 18:12:50.552657 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:50.552633 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k"] Sep 17 18:12:52.149932 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:52.149898 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" path="/var/lib/kubelet/pods/35f8c37d-87c1-480a-84fa-e4d7c4dae3cb/volumes" Sep 17 18:12:55.126769 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:12:55.126730 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-c1efc029-kserve-854f48dbc9-frv7k" podUID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 17 18:13:00.466266 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466180 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2"] Sep 17 18:13:00.466711 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466694 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" containerName="main" Sep 17 18:13:00.466760 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466716 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" containerName="main" Sep 17 18:13:00.466760 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466745 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f7ba22-5739-47f3-a14d-43b35c8b46ee" containerName="main" Sep 17 18:13:00.466760 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466754 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f7ba22-5739-47f3-a14d-43b35c8b46ee" containerName="main" Sep 17 18:13:00.466866 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466835 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f7ba22-5739-47f3-a14d-43b35c8b46ee" containerName="main" Sep 17 18:13:00.466866 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.466851 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f8c37d-87c1-480a-84fa-e4d7c4dae3cb" containerName="main" Sep 17 18:13:00.471612 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.471587 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.481158 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.481133 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2"] Sep 17 18:13:00.576250 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.576212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7c2335-ebe2-465b-a414-f718c740f6eb-metrics-certs\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.576433 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.576286 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vn5w\" (UniqueName: \"kubernetes.io/projected/2f7c2335-ebe2-465b-a414-f718c740f6eb-kube-api-access-9vn5w\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.576433 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.576309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c2335-ebe2-465b-a414-f718c740f6eb-openshift-service-ca\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.576433 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.576333 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7c2335-ebe2-465b-a414-f718c740f6eb-maas-api-tls\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.677540 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.677502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7c2335-ebe2-465b-a414-f718c740f6eb-maas-api-tls\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.677737 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.677608 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7c2335-ebe2-465b-a414-f718c740f6eb-metrics-certs\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.677737 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.677666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vn5w\" (UniqueName: \"kubernetes.io/projected/2f7c2335-ebe2-465b-a414-f718c740f6eb-kube-api-access-9vn5w\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.677737 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.677692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c2335-ebe2-465b-a414-f718c740f6eb-openshift-service-ca\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.678367 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.678345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7c2335-ebe2-465b-a414-f718c740f6eb-openshift-service-ca\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.680271 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.680244 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7c2335-ebe2-465b-a414-f718c740f6eb-metrics-certs\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.680378 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.680244 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7c2335-ebe2-465b-a414-f718c740f6eb-maas-api-tls\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.685592 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.685572 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vn5w\" (UniqueName: \"kubernetes.io/projected/2f7c2335-ebe2-465b-a414-f718c740f6eb-kube-api-access-9vn5w\") pod \"maas-api-65c5475bc5-z48p2\" (UID: \"2f7c2335-ebe2-465b-a414-f718c740f6eb\") " pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:00.784115 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:00.784026 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:01.053919 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:01.053889 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2"] Sep 17 18:13:01.056661 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:13:01.056631 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7c2335_ebe2_465b_a414_f718c740f6eb.slice/crio-358aeda9f8df625185d23ba0d51dffe8b5fed4c5e2b0a767e1629ca829252ff6 WatchSource:0}: Error finding container 358aeda9f8df625185d23ba0d51dffe8b5fed4c5e2b0a767e1629ca829252ff6: Status 404 returned error can't find the container with id 358aeda9f8df625185d23ba0d51dffe8b5fed4c5e2b0a767e1629ca829252ff6 Sep 17 18:13:01.283447 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:01.283408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" event={"ID":"2f7c2335-ebe2-465b-a414-f718c740f6eb","Type":"ContainerStarted","Data":"358aeda9f8df625185d23ba0d51dffe8b5fed4c5e2b0a767e1629ca829252ff6"} Sep 17 18:13:02.290144 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:02.290105 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" event={"ID":"2f7c2335-ebe2-465b-a414-f718c740f6eb","Type":"ContainerStarted","Data":"b4f495001350b759ced81d83e1dcea0e36731d69317604cb20e6a34a957e30d3"} Sep 17 18:13:02.290536 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:02.290202 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:02.308911 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:02.308855 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" podStartSLOduration=1.958346992 podStartE2EDuration="2.308841345s" totalImagesPullingTime="350.494353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:02.307869049 +0000 UTC m=+2518.870033908" watchObservedRunningTime="2026-09-17 18:13:02.308841345 +0000 UTC m=+2518.871006247" Sep 17 18:13:08.300644 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:08.300617 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65c5475bc5-z48p2" Sep 17 18:13:08.349722 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:08.349686 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2"] Sep 17 18:13:08.349972 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:08.349930 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" podUID="9e6fb890-676a-4ec4-9e85-55864bf63eff" containerName="maas-api" containerID="cri-o://a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb" gracePeriod=30 Sep 17 18:13:09.596237 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.596213 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 18:13:09.761242 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.761157 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6fb890-676a-4ec4-9e85-55864bf63eff-openshift-service-ca\") pod \"9e6fb890-676a-4ec4-9e85-55864bf63eff\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " Sep 17 18:13:09.761242 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.761188 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlw2l\" (UniqueName: \"kubernetes.io/projected/9e6fb890-676a-4ec4-9e85-55864bf63eff-kube-api-access-mlw2l\") pod \"9e6fb890-676a-4ec4-9e85-55864bf63eff\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " Sep 17 18:13:09.761242 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.761220 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-maas-api-tls\") pod \"9e6fb890-676a-4ec4-9e85-55864bf63eff\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " Sep 17 18:13:09.761511 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.761280 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs\") pod \"9e6fb890-676a-4ec4-9e85-55864bf63eff\" (UID: \"9e6fb890-676a-4ec4-9e85-55864bf63eff\") " Sep 17 18:13:09.761574 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.761524 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6fb890-676a-4ec4-9e85-55864bf63eff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e6fb890-676a-4ec4-9e85-55864bf63eff" (UID: "9e6fb890-676a-4ec4-9e85-55864bf63eff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:13:09.763382 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.763346 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e6fb890-676a-4ec4-9e85-55864bf63eff" (UID: "9e6fb890-676a-4ec4-9e85-55864bf63eff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:13:09.763523 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.763388 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6fb890-676a-4ec4-9e85-55864bf63eff-kube-api-access-mlw2l" (OuterVolumeSpecName: "kube-api-access-mlw2l") pod "9e6fb890-676a-4ec4-9e85-55864bf63eff" (UID: "9e6fb890-676a-4ec4-9e85-55864bf63eff"). InnerVolumeSpecName "kube-api-access-mlw2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:13:09.763523 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.763439 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e6fb890-676a-4ec4-9e85-55864bf63eff" (UID: "9e6fb890-676a-4ec4-9e85-55864bf63eff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:13:09.862206 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.862169 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6fb890-676a-4ec4-9e85-55864bf63eff-openshift-service-ca\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:13:09.862206 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.862202 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlw2l\" (UniqueName: \"kubernetes.io/projected/9e6fb890-676a-4ec4-9e85-55864bf63eff-kube-api-access-mlw2l\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:13:09.862407 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.862215 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-maas-api-tls\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:13:09.862407 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:09.862229 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6fb890-676a-4ec4-9e85-55864bf63eff-metrics-certs\") on node \"ip-10-0-142-117.ec2.internal\" DevicePath \"\"" Sep 17 18:13:10.328495 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.328460 2566 generic.go:356] "Generic (PLEG): container finished" podID="9e6fb890-676a-4ec4-9e85-55864bf63eff" containerID="a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb" exitCode=0 Sep 17 18:13:10.328706 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.328508 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" Sep 17 18:13:10.328706 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.328550 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" event={"ID":"9e6fb890-676a-4ec4-9e85-55864bf63eff","Type":"ContainerDied","Data":"a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb"} Sep 17 18:13:10.328706 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.328609 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2" event={"ID":"9e6fb890-676a-4ec4-9e85-55864bf63eff","Type":"ContainerDied","Data":"d9222c8e1ab9d34e6bb815f412886e6250622a16f632b089ac9295a813e88dab"} Sep 17 18:13:10.328706 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.328639 2566 scope.go:117] "RemoveContainer" containerID="a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb" Sep 17 18:13:10.337354 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.337337 2566 scope.go:117] "RemoveContainer" containerID="a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb" Sep 17 18:13:10.337647 ip-10-0-142-117 kubenswrapper[2566]: E0917 18:13:10.337627 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb\": container with ID starting with a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb not found: ID does not exist" containerID="a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb" Sep 17 18:13:10.337719 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.337656 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb"} err="failed to get container status \"a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb\": rpc error: code = NotFound desc = could not find container \"a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb\": container with ID starting with a5147e255f64a5108a7823505dce55906b636d431f3448550a484f4f860acbcb not found: ID does not exist" Sep 17 18:13:10.344912 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.344890 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2"] Sep 17 18:13:10.349037 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:10.349015 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c99d599cf-5ngv2"] Sep 17 18:13:12.149473 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:12.149442 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e6fb890-676a-4ec4-9e85-55864bf63eff" path="/var/lib/kubelet/pods/9e6fb890-676a-4ec4-9e85-55864bf63eff/volumes" Sep 17 18:13:28.709336 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:28.709303 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dxldr_0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c/manager/0.log" Sep 17 18:13:28.828728 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:28.828695 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-j7p4c_65dd8a4a-3c8b-4a84-856e-902773c15f8c/manager/0.log" Sep 17 18:13:28.947235 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:28.947203 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b777b4495-8mxgh_d5471202-2060-45dd-8fa4-c330b636b73d/manager/0.log" Sep 17 18:13:29.058546 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:29.058521 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n9ptj_1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4/server/0.log" Sep 17 18:13:29.537925 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:29.537832 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-584f5b8894-qvzwm_34a1d4cf-3914-42ab-bfbf-93e21ced2168/copy-manifests/0.log" Sep 17 18:13:29.544009 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:29.543985 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-584f5b8894-qvzwm_34a1d4cf-3914-42ab-bfbf-93e21ced2168/manager/0.log" Sep 17 18:13:29.870035 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:29.870002 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65c5475bc5-z48p2_2f7c2335-ebe2-465b-a414-f718c740f6eb/maas-api/0.log" Sep 17 18:13:30.304702 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:30.304665 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rrk87_1db38631-3f8d-4008-afda-3ce44c78112d/postgres/0.log" Sep 17 18:13:31.183813 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:31.183786 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-z65qv_3d36aa81-6545-4d9b-98e5-babd8336ea87/manager/0.log" Sep 17 18:13:31.290582 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:31.290556 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-zgws5_31051d3c-f712-49f7-b929-af039212ce93/manager/0.log" Sep 17 18:13:31.525175 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:31.525100 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-v6czp_6308ccf3-d7f0-4e23-9de7-8f3c52251a62/manager/0.log" Sep 17 18:13:31.759198 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:31.759171 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-j5g87_ac835e77-2abe-4e4b-97f0-e2a1235fdbd4/manager/0.log" Sep 17 18:13:32.420048 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:32.420014 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5499794fdf-nt54m_7c0489bd-a206-419b-88eb-8de14a1c9595/kube-auth-proxy/0.log" Sep 17 18:13:32.643257 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:32.643204 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-98qth_93720089-403a-41aa-ae26-d739aee03bc3/payload-pre-processing/0.log" Sep 17 18:13:33.217838 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:33.217799 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c6zn5_0e1dfeab-8ea2-4bbf-bc25-f72e83317eb7/main/0.log" Sep 17 18:13:33.324179 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:33.324153 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6tqgx_1263642c-7d0e-47ec-9625-2057f27bdfbb/main/0.log" Sep 17 18:13:33.751185 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:33.751152 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jgxwr_1d0773dd-19eb-47cd-929a-a4a691fef141/main/0.log" Sep 17 18:13:33.863239 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:33.863211 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2s7pk_58c000c9-0b8d-4d7c-9c5a-3a8fb986f549/main/0.log" Sep 17 18:13:42.326195 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:42.326166 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qvfw4_d889d683-785d-40d7-a042-9e95c805b529/global-pull-secret-syncer/0.log" Sep 17 18:13:42.415700 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:42.415672 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t28t7_6d360eb6-6699-4278-95fc-55d7f937ff4c/konnectivity-agent/0.log" Sep 17 18:13:42.510587 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:42.510557 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-117.ec2.internal_bf78f1345f70a888653325ba64aacb3d/haproxy/0.log" Sep 17 18:13:47.652736 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:47.652703 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-z65qv_3d36aa81-6545-4d9b-98e5-babd8336ea87/manager/0.log" Sep 17 18:13:47.677695 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:47.677659 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-zgws5_31051d3c-f712-49f7-b929-af039212ce93/manager/0.log" Sep 17 18:13:47.759683 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:47.759649 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-v6czp_6308ccf3-d7f0-4e23-9de7-8f3c52251a62/manager/0.log" Sep 17 18:13:47.875607 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:47.875569 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-j5g87_ac835e77-2abe-4e4b-97f0-e2a1235fdbd4/manager/0.log" Sep 17 18:13:49.796919 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:49.796882 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hs8h_6cc72b41-3410-4e68-8c43-c7f487adde2d/node-exporter/0.log" Sep 17 18:13:49.819358 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:49.819332 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hs8h_6cc72b41-3410-4e68-8c43-c7f487adde2d/kube-rbac-proxy/0.log" Sep 17 18:13:49.847015 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:49.846994 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hs8h_6cc72b41-3410-4e68-8c43-c7f487adde2d/init-textfile/0.log" Sep 17 18:13:52.753705 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.753672 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb"] Sep 17 18:13:52.754095 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.754059 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e6fb890-676a-4ec4-9e85-55864bf63eff" containerName="maas-api" Sep 17 18:13:52.754095 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.754071 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e6fb890-676a-4ec4-9e85-55864bf63eff" containerName="maas-api" Sep 17 18:13:52.754175 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.754131 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e6fb890-676a-4ec4-9e85-55864bf63eff" containerName="maas-api" Sep 17 18:13:52.757094 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.757075 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.759165 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.759144 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kb2r\"/\"openshift-service-ca.crt\"" Sep 17 18:13:52.759282 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.759267 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8kb2r\"/\"default-dockercfg-dkft9\"" Sep 17 18:13:52.760058 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.760042 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kb2r\"/\"kube-root-ca.crt\"" Sep 17 18:13:52.764487 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.764465 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb"] Sep 17 18:13:52.818695 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.818665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brsfx\" (UniqueName: \"kubernetes.io/projected/71bf59a9-c744-4109-90a1-c3f909ffd1d2-kube-api-access-brsfx\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.818695 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.818696 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-sys\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.818868 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.818732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-proc\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.818868 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.818749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-podres\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.818868 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.818770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-lib-modules\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919553 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919526 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-proc\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919558 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-podres\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-lib-modules\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919632 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brsfx\" (UniqueName: \"kubernetes.io/projected/71bf59a9-c744-4109-90a1-c3f909ffd1d2-kube-api-access-brsfx\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-proc\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919658 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-sys\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-podres\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.919770 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919733 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-sys\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.920074 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.919769 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71bf59a9-c744-4109-90a1-c3f909ffd1d2-lib-modules\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:52.927379 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:52.927359 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brsfx\" (UniqueName: \"kubernetes.io/projected/71bf59a9-c744-4109-90a1-c3f909ffd1d2-kube-api-access-brsfx\") pod \"perf-node-gather-daemonset-292mb\" (UID: \"71bf59a9-c744-4109-90a1-c3f909ffd1d2\") " pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:53.067184 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:53.067164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:53.335503 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:53.335476 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb"] Sep 17 18:13:53.337516 ip-10-0-142-117 kubenswrapper[2566]: W0917 18:13:53.337486 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod71bf59a9_c744_4109_90a1_c3f909ffd1d2.slice/crio-96ca08dfd94ff72b6f5be90d6fb2dd44b79b413ed19a3fc1a057b953ac74ad2e WatchSource:0}: Error finding container 96ca08dfd94ff72b6f5be90d6fb2dd44b79b413ed19a3fc1a057b953ac74ad2e: Status 404 returned error can't find the container with id 96ca08dfd94ff72b6f5be90d6fb2dd44b79b413ed19a3fc1a057b953ac74ad2e Sep 17 18:13:53.521791 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:53.521757 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" event={"ID":"71bf59a9-c744-4109-90a1-c3f909ffd1d2","Type":"ContainerStarted","Data":"dc1e8ba66a1e4b6367b56b9d9ad4dd2d4a62711d242c74c53382f1c74f531ea1"} Sep 17 18:13:53.521901 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:53.521815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" event={"ID":"71bf59a9-c744-4109-90a1-c3f909ffd1d2","Type":"ContainerStarted","Data":"96ca08dfd94ff72b6f5be90d6fb2dd44b79b413ed19a3fc1a057b953ac74ad2e"} Sep 17 18:13:53.521976 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:53.521898 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:13:53.539631 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:53.539566 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" podStartSLOduration=1.539550328 podStartE2EDuration="1.539550328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:53.539148862 +0000 UTC m=+2570.101313726" watchObservedRunningTime="2026-09-17 18:13:53.539550328 +0000 UTC m=+2570.101715188" Sep 17 18:13:55.122922 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:55.122892 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wm2jb_84917d34-2512-43fa-86d4-e7c00104371d/dns/0.log" Sep 17 18:13:55.143235 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:55.143204 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wm2jb_84917d34-2512-43fa-86d4-e7c00104371d/kube-rbac-proxy/0.log" Sep 17 18:13:55.211461 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:55.211429 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vppd9_eb353df6-c110-4406-9efb-e29e653be6dc/dns-node-resolver/0.log" Sep 17 18:13:55.762308 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:55.762281 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7g2lv_2b69f4eb-f959-4d3c-9331-3a8ddf0af363/node-ca/0.log" Sep 17 18:13:57.236285 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:57.236258 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5499794fdf-nt54m_7c0489bd-a206-419b-88eb-8de14a1c9595/kube-auth-proxy/0.log" Sep 17 18:13:57.308995 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:57.308969 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-98qth_93720089-403a-41aa-ae26-d739aee03bc3/payload-pre-processing/0.log" Sep 17 18:13:58.289023 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:58.288997 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrssb_33f130cc-b73a-4f66-82b6-0b687aa48686/serve-healthcheck-canary/0.log" Sep 17 18:13:58.782529 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:58.782501 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g8qdz_162124ac-91f3-43bd-b7b1-018b7616814f/kube-rbac-proxy/0.log" Sep 17 18:13:58.800323 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:58.800303 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g8qdz_162124ac-91f3-43bd-b7b1-018b7616814f/exporter/0.log" Sep 17 18:13:58.819000 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:58.818978 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g8qdz_162124ac-91f3-43bd-b7b1-018b7616814f/extractor/0.log" Sep 17 18:13:59.534999 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:13:59.534971 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8kb2r/perf-node-gather-daemonset-292mb" Sep 17 18:14:01.064989 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:01.064945 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dxldr_0c91bdae-c4aa-41ab-9dca-5a4fcb09de9c/manager/0.log" Sep 17 18:14:01.530785 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:01.530695 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-j7p4c_65dd8a4a-3c8b-4a84-856e-902773c15f8c/manager/0.log" Sep 17 18:14:01.600366 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:01.600332 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b777b4495-8mxgh_d5471202-2060-45dd-8fa4-c330b636b73d/manager/0.log" Sep 17 18:14:01.618871 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:01.618844 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n9ptj_1c1d7ab9-7f46-4896-b9f4-97ed05a5ecd4/server/0.log" Sep 17 18:14:01.895396 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:01.895362 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-584f5b8894-qvzwm_34a1d4cf-3914-42ab-bfbf-93e21ced2168/manager/0.log" Sep 17 18:14:01.917638 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:01.917610 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-584f5b8894-qvzwm_34a1d4cf-3914-42ab-bfbf-93e21ced2168/copy-manifests/0.log" Sep 17 18:14:03.325977 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:03.325929 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5499794fdf-nt54m_7c0489bd-a206-419b-88eb-8de14a1c9595/kube-auth-proxy/0.log" Sep 17 18:14:03.379617 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:03.379579 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-98qth_93720089-403a-41aa-ae26-d739aee03bc3/payload-pre-processing/0.log" Sep 17 18:14:05.548099 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:05.548016 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5499794fdf-nt54m_7c0489bd-a206-419b-88eb-8de14a1c9595/kube-auth-proxy/0.log" Sep 17 18:14:05.615043 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:05.615016 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-98qth_93720089-403a-41aa-ae26-d739aee03bc3/payload-pre-processing/0.log" Sep 17 18:14:10.179240 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.179213 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/kube-multus-additional-cni-plugins/0.log" Sep 17 18:14:10.199420 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.199397 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/egress-router-binary-copy/0.log" Sep 17 18:14:10.219139 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.219121 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/cni-plugins/0.log" Sep 17 18:14:10.238238 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.238210 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/bond-cni-plugin/0.log" Sep 17 18:14:10.256752 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.256731 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/routeoverride-cni/0.log" Sep 17 18:14:10.274740 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.274723 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/whereabouts-cni-bincopy/0.log" Sep 17 18:14:10.294696 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.294675 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q85q4_d317ebb7-0fa0-4b76-a2f0-97e5e0343bbe/whereabouts-cni/0.log" Sep 17 18:14:10.595971 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.595926 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s9jkx_6cf9219f-4e4a-482d-a394-b235e5723aed/kube-multus/0.log" Sep 17 18:14:10.619858 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.619832 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lbj98_fc9a2b4e-2ae6-496a-9db8-abfbed21ab09/network-metrics-daemon/0.log" Sep 17 18:14:10.633876 ip-10-0-142-117 kubenswrapper[2566]: I0917 18:14:10.633846 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lbj98_fc9a2b4e-2ae6-496a-9db8-abfbed21ab09/kube-rbac-proxy/0.log"