Sep 18 19:59:03.636253 ip-10-0-135-232 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 18 19:59:03.636263 ip-10-0-135-232 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 18 19:59:03.636270 ip-10-0-135-232 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 18 19:59:03.636511 ip-10-0-135-232 systemd[1]: Failed to start Kubernetes Kubelet. Sep 18 19:59:13.775536 ip-10-0-135-232 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 18 19:59:13.775551 ip-10-0-135-232 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d3c690d4812e4e25ad9f70881c855a9a -- Sep 18 20:01:19.553607 ip-10-0-135-232 systemd[1]: Starting Kubernetes Kubelet... Sep 18 20:01:22.991050 ip-10-0-135-232 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 20:01:22.991050 ip-10-0-135-232 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 20:01:22.991050 ip-10-0-135-232 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 20:01:22.991050 ip-10-0-135-232 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 20:01:22.991050 ip-10-0-135-232 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 20:01:22.991855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.991125 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 20:01:22.994919 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994900 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 20:01:22.994919 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994918 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994922 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994925 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994931 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994934 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994938 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994941 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994944 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994946 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994949 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994952 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994955 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994957 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994960 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994962 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994973 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994976 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994981 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994985 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 20:01:22.994987 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994988 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994991 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994994 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.994997 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995001 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995004 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995007 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995009 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995012 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995015 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995018 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995021 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995024 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995027 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995029 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995032 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995034 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995037 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995040 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995043 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 20:01:22.995449 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995045 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995048 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995051 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995053 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995056 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995058 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995061 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995064 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995066 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995068 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995071 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995073 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995076 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995078 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995083 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995085 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995089 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995092 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995095 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 20:01:22.995932 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995097 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995101 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995105 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995108 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995111 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995114 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995118 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995120 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995123 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995125 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995131 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995134 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995137 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995140 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995143 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995145 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995148 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995151 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995154 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 20:01:22.996444 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995156 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995159 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995161 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995164 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995167 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995169 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995171 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995174 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995177 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995180 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995183 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995185 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995188 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995190 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995193 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995196 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995198 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995200 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995203 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995206 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 20:01:22.996898 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995208 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995964 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995973 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995976 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995980 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995983 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995986 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995989 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995993 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995996 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.995999 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996002 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996005 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996007 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996010 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996012 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996015 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996017 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996020 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 20:01:22.997398 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996023 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996027 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996030 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996034 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996037 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996040 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996042 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996045 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996048 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996050 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996053 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996055 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996058 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996060 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996063 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996065 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996068 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996071 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996073 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996076 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 20:01:22.997889 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996079 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996081 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996086 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996089 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996092 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996094 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996097 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996100 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996102 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996105 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996109 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996112 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996115 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996118 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996120 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996124 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996127 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996129 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996132 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 20:01:22.998393 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996135 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996137 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996140 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996142 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996145 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996148 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996150 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996153 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996155 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996158 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996160 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996163 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996165 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996168 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996171 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996174 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996176 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996179 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996181 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996184 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 20:01:22.998859 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996186 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996189 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996191 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996194 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996196 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996200 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996203 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996205 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996208 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996210 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996213 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996215 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996218 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996220 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996223 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996225 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996228 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996230 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996233 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996235 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 20:01:22.999361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996238 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.996240 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997386 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997393 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997397 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997400 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997404 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997407 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997410 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997414 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997416 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997419 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997422 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997425 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997427 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997430 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997433 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997437 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997439 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997442 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 20:01:22.999851 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997445 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997447 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997450 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997452 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997455 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997457 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997460 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997462 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997465 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997468 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997470 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997473 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997475 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997478 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997480 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997483 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997485 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997489 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997492 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 20:01:23.000364 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997496 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997499 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997502 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997505 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997508 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997510 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997513 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997515 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997518 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997520 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997523 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997527 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997529 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997532 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997534 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997537 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997540 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997542 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997545 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997547 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 20:01:23.000824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997550 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997552 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997555 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997559 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997562 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997564 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997567 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997570 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997572 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997575 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997577 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997580 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997582 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997587 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997594 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997597 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997600 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997602 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997605 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 20:01:23.001359 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997608 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997610 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997613 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997616 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997619 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997621 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997624 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997626 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997629 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997631 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997634 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997637 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997640 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997642 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997645 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997648 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997650 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997652 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997655 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 20:01:23.001823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997657 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997660 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997663 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.997665 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997742 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997785 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997795 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997799 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997810 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997813 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997818 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997822 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997825 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997828 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997832 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997836 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997842 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997845 2559 flags.go:64] FLAG: --cgroup-root="" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997849 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997852 2559 flags.go:64] FLAG: --client-ca-file="" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997855 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997858 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997862 2559 flags.go:64] FLAG: --cluster-domain="" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997865 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 20:01:23.002298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997868 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997871 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997875 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997879 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997883 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997886 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997889 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997892 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997895 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997898 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997901 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997904 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997914 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997917 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997920 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997923 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997928 2559 flags.go:64] FLAG: --enable-server="true" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997932 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997937 2559 flags.go:64] FLAG: --event-burst="100" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997940 2559 flags.go:64] FLAG: --event-qps="50" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997943 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997946 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997949 2559 flags.go:64] FLAG: --eviction-hard="" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997953 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 20:01:23.002879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997957 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997960 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997964 2559 flags.go:64] FLAG: --eviction-soft="" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997967 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997969 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997973 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997976 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997979 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997982 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997984 2559 flags.go:64] FLAG: --feature-gates="" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997988 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997991 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997994 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.997998 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998001 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998004 2559 flags.go:64] FLAG: --help="false" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998007 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998011 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998014 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998017 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998020 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998023 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998026 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 20:01:23.003467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998029 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998033 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998036 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998040 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998043 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998046 2559 flags.go:64] FLAG: --kube-reserved="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998049 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998051 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998054 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998059 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998062 2559 flags.go:64] FLAG: --lock-file="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998065 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998068 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998071 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998077 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998080 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998082 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998085 2559 flags.go:64] FLAG: --logging-format="text" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998088 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998092 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998095 2559 flags.go:64] FLAG: --manifest-url="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998098 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998102 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998105 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998109 2559 flags.go:64] FLAG: --max-pods="110" Sep 18 20:01:23.004045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998112 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998115 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998118 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998121 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998125 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998127 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998130 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998138 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998142 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998145 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998148 2559 flags.go:64] FLAG: --pod-cidr="" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998151 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998157 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998160 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998163 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998166 2559 flags.go:64] FLAG: --port="10250" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998169 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998173 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00553d4cb53814d12" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998177 2559 flags.go:64] FLAG: --qos-reserved="" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998179 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998182 2559 flags.go:64] FLAG: --register-node="true" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998185 2559 flags.go:64] FLAG: --register-with-taints="" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998189 2559 flags.go:64] FLAG: --registry-burst="10" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998192 2559 flags.go:64] FLAG: --registry-qps="5" Sep 18 20:01:23.004767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998194 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998197 2559 flags.go:64] FLAG: --reserved-memory="" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998201 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998204 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998208 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998210 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998213 2559 flags.go:64] FLAG: --runonce="false" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998217 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998220 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998223 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998226 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998228 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998231 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998235 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998238 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998241 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998245 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998248 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998251 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998254 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998258 2559 flags.go:64] FLAG: --system-cgroups="" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998261 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998266 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998269 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998285 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 20:01:23.005380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998291 2559 flags.go:64] FLAG: --tls-min-version="" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998293 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998296 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998299 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998302 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998305 2559 flags.go:64] FLAG: --v="2" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998309 2559 flags.go:64] FLAG: --version="false" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998313 2559 flags.go:64] FLAG: --vmodule="" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998317 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.998320 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998469 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998472 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998476 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998479 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998481 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998484 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998486 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998489 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998492 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998494 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998497 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998499 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 20:01:23.005993 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998510 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998514 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998517 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998520 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998523 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998525 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998529 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998532 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998535 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998537 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998540 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998543 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998545 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998548 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998550 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998553 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998556 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998558 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998561 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998563 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 20:01:23.006760 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998566 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998568 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998571 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998574 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998576 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998579 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998582 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998584 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998587 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998589 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998591 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998594 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998597 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998601 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998604 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998606 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998609 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998612 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998614 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 20:01:23.007345 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998618 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998620 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998623 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998626 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998629 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998631 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998634 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998638 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998641 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998644 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998647 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998650 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998652 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998655 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998658 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998660 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998663 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998666 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998669 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 20:01:23.007823 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998671 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998674 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998676 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998679 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998681 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998684 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998686 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998690 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998693 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998695 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998697 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998700 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998703 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998705 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998708 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998711 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998714 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998716 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998719 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998721 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 20:01:23.008338 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998724 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998726 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998729 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998733 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998737 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998740 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998742 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998746 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:22.998750 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:22.999197 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.007775 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.007792 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.007808 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.007814 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 20:01:23.008831 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007869 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007874 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007878 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007881 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007884 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007888 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007891 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007894 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007897 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007900 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007903 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007907 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007911 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007914 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007917 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007920 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007923 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007925 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007928 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 20:01:23.009191 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007931 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007933 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007936 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007939 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007941 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007944 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007947 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007949 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007953 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007956 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007959 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007962 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007965 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007968 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007971 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007974 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007976 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007979 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007982 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 20:01:23.009773 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007984 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007987 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007990 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007994 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.007998 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008001 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008004 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008006 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008009 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008012 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008014 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008017 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008020 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008023 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008025 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008028 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008030 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008033 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008036 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008039 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 20:01:23.010228 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008042 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008045 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008048 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008051 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008054 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008057 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008059 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008063 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008065 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008068 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008071 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008075 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008078 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008081 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008085 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008088 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008091 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008093 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008096 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 20:01:23.010753 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008099 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008102 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008105 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008108 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008111 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008114 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008116 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008119 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008122 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008124 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008127 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008129 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008132 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008134 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008137 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008140 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008143 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008146 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008149 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008152 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 20:01:23.011226 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008154 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008157 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.008162 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008293 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008299 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008302 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008305 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008308 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008311 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008314 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008316 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008321 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008325 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008328 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 20:01:23.011712 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008331 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008334 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008337 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008340 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008344 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008347 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008350 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008353 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008355 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008358 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008361 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008363 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008366 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008368 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008372 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008375 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008377 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008380 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008383 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008385 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 20:01:23.012078 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008388 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008391 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008394 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008396 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008399 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008402 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008404 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008407 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008409 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008412 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008415 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008417 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008420 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008422 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008426 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008428 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008432 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008435 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008438 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008441 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 20:01:23.012614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008444 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008447 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008450 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008453 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008456 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008458 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008461 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008465 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008468 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008471 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008474 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008476 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008480 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008483 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008486 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008488 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008491 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008494 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008496 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 20:01:23.013115 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008499 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008502 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008504 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008507 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008509 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008512 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008514 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008517 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008520 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008523 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008525 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008528 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008530 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008533 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008535 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008538 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008541 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008543 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008546 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 20:01:23.013614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008549 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008551 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008554 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008557 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008559 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008562 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008565 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008567 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008570 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:23.008572 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.008577 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.008690 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.010801 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.011765 2559 server.go:1019] "Starting client certificate rotation" Sep 18 20:01:23.014119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.011861 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 20:01:23.014539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.012695 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 20:01:23.014539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.014010 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 20:01:23.015204 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.015186 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 20:01:23.026092 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.026075 2559 log.go:25] "Validated CRI v1 runtime API" Sep 18 20:01:23.027843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.027831 2559 log.go:25] "Validated CRI v1 image API" Sep 18 20:01:23.028656 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.028636 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 20:01:23.031794 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.031773 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c43d040-c960-45cc-a0ed-318d6be34faa:/dev/nvme0n1p4 8e9c4700-7206-4db3-a72d-9804879bbc37:/dev/nvme0n1p3] Sep 18 20:01:23.031862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.031792 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 20:01:23.037652 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.037521 2559 manager.go:216] Machine: {Timestamp:2026-09-18 20:01:23.037001901 +0000 UTC m=+2.087362339 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119367 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b5d0c3c44142eedaf4c948225223 SystemUUID:ec20b5d0-c3c4-4142-eeda-f4c948225223 BootID:d3c690d4-812e-4e25-ad9f-70881c855a9a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:03:de:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:03:de:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:2a:be:68:a6:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 20:01:23.037652 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.037650 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 20:01:23.037805 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.037787 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 20:01:23.039325 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.039299 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 20:01:23.039490 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.039327 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 20:01:23.040787 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.040770 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 20:01:23.040872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.040795 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 20:01:23.040872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.040807 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 20:01:23.040872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.040824 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 20:01:23.041016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.040918 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 20:01:23.043726 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043709 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 18 20:01:23.043859 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043849 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 20:01:23.043900 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043886 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 18 20:01:23.043900 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043896 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 20:01:23.043959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043912 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 20:01:23.043959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043923 2559 kubelet.go:398] "Adding apiserver pod source" Sep 18 20:01:23.043959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.043944 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 20:01:23.044412 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.044401 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 20:01:23.044461 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.044420 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 20:01:23.046904 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.046888 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 20:01:23.049467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.049454 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 20:01:23.049514 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.049485 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 20:01:23.050138 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050116 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 20:01:23.050659 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050643 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050670 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050680 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050688 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050702 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050712 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050732 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 20:01:23.050743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050740 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 20:01:23.050986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050750 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 20:01:23.050986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050758 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 20:01:23.050986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050772 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 20:01:23.050986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050781 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 20:01:23.050986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050822 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 20:01:23.050986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.050831 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 20:01:23.058242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.058225 2559 server.go:1268] "Started kubelet" Sep 18 20:01:23.058327 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.058294 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 20:01:23.058525 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.058473 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 20:01:23.058564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.058548 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 20:01:23.059800 ip-10-0-135-232 systemd[1]: Started Kubernetes Kubelet. Sep 18 20:01:23.060421 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.059834 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 20:01:23.060975 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.060819 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 18 20:01:23.061887 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.061868 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 20:01:23.062072 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.062053 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 20:01:23.062173 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.062058 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 20:01:23.065443 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.065424 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 20:01:23.066058 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.066044 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 20:01:23.066149 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.066135 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 20:01:23.066201 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.066160 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.066614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.066596 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 20:01:23.066691 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.066618 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 20:01:23.066691 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.066594 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 20:01:23.067304 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067288 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 18 20:01:23.067304 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067304 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 18 20:01:23.067438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067424 2559 factory.go:55] Registering systemd factory Sep 18 20:01:23.067493 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067474 2559 factory.go:223] Registration of the systemd container factory successfully Sep 18 20:01:23.067709 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067694 2559 factory.go:153] Registering CRI-O factory Sep 18 20:01:23.067709 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067709 2559 factory.go:223] Registration of the crio container factory successfully Sep 18 20:01:23.067826 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067757 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 20:01:23.067826 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067776 2559 factory.go:103] Registering Raw factory Sep 18 20:01:23.067826 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.067793 2559 manager.go:1196] Started watching for new ooms in manager Sep 18 20:01:23.068164 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.068135 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 20:01:23.068288 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.068255 2559 manager.go:349] Starting recovery of all containers Sep 18 20:01:23.068365 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.068264 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 20:01:23.069737 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.069652 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddbnp" Sep 18 20:01:23.086426 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.086386 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddbnp" Sep 18 20:01:23.094382 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.094363 2559 manager.go:354] Recovery completed Sep 18 20:01:23.098783 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.098770 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.101022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101003 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.101080 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101040 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.101080 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101051 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.101621 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101607 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 20:01:23.101621 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101618 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.101621 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101620 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 20:01:23.101802 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.101638 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 18 20:01:23.102312 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.102296 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.102379 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.102326 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.102379 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.102340 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.102903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.102891 2559 policy_none.go:49] "None policy: Start" Sep 18 20:01:23.102946 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.102912 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 20:01:23.102946 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.102922 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 20:01:23.103976 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.103960 2559 policy_none.go:47] "Start" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.144845 2559 manager.go:337] "Starting Device Plugin manager" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.144881 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.144892 2559 server.go:87] "Starting device plugin registration server" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.145185 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.145209 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.145298 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.145398 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.145409 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.145905 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 20:01:23.146316 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.145954 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.175806 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.175769 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 20:01:23.176994 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.176977 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 20:01:23.177145 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.177036 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 20:01:23.177145 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.177068 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 20:01:23.177145 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.177112 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 20:01:23.179477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.179451 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 20:01:23.245584 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.245490 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.246504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.246486 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.246593 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.246517 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.246593 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.246529 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.246593 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.246553 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.256168 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.256152 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.256212 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.256176 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-232.ec2.internal\": node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.273148 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.273129 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.278080 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.278054 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal"] Sep 18 20:01:23.278137 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.278128 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.279635 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.279609 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.279702 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.279645 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.279702 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.279655 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.280841 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.280822 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.280951 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.280938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.280988 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.280972 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.281617 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.281595 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.281718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.281618 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.281718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.281644 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.281718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.281648 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.281718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.281656 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.281718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.281674 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.282963 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.282941 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.283071 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.282985 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 20:01:23.283757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.283743 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientMemory" Sep 18 20:01:23.283834 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.283767 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 20:01:23.283834 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.283780 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeHasSufficientPID" Sep 18 20:01:23.308062 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.308036 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-232.ec2.internal\" not found" node="ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.312842 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.312823 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-232.ec2.internal\" not found" node="ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.368367 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.368320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a5f3f19b1eab7558136f3b3fab9b6d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal\" (UID: \"0a5f3f19b1eab7558136f3b3fab9b6d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.368367 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.368367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f3f19b1eab7558136f3b3fab9b6d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal\" (UID: \"0a5f3f19b1eab7558136f3b3fab9b6d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.368367 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.368385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ffd3a0f2bc2beb85186ad42f65a5756-config\") pod \"kube-apiserver-proxy-ip-10-0-135-232.ec2.internal\" (UID: \"5ffd3a0f2bc2beb85186ad42f65a5756\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.373369 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.373347 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.468902 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.468869 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a5f3f19b1eab7558136f3b3fab9b6d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal\" (UID: \"0a5f3f19b1eab7558136f3b3fab9b6d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.468902 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.468839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a5f3f19b1eab7558136f3b3fab9b6d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal\" (UID: \"0a5f3f19b1eab7558136f3b3fab9b6d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.469105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.468943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f3f19b1eab7558136f3b3fab9b6d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal\" (UID: \"0a5f3f19b1eab7558136f3b3fab9b6d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.469105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.468971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ffd3a0f2bc2beb85186ad42f65a5756-config\") pod \"kube-apiserver-proxy-ip-10-0-135-232.ec2.internal\" (UID: \"5ffd3a0f2bc2beb85186ad42f65a5756\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.469105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.469029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f3f19b1eab7558136f3b3fab9b6d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal\" (UID: \"0a5f3f19b1eab7558136f3b3fab9b6d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.469105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.469042 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ffd3a0f2bc2beb85186ad42f65a5756-config\") pod \"kube-apiserver-proxy-ip-10-0-135-232.ec2.internal\" (UID: \"5ffd3a0f2bc2beb85186ad42f65a5756\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.473950 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.473918 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.574797 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.574731 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.611936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.611899 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.616800 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.616598 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" Sep 18 20:01:23.675249 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.675060 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.775635 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.775604 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.872572 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:23.872515 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 20:01:23.876638 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.876622 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:23.977283 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:23.977250 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.011389 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.011372 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 20:01:24.011794 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.011497 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 20:01:24.011794 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.011534 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 20:01:24.077773 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:24.077744 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.091984 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.091943 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:56:23 +0000 UTC" deadline="2028-05-01 15:05:56.738541935 +0000 UTC" Sep 18 20:01:24.091984 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.091981 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14179h4m32.646563908s" Sep 18 20:01:24.178650 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:24.178620 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.228336 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.228303 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 20:01:24.275003 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:24.274967 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffd3a0f2bc2beb85186ad42f65a5756.slice/crio-9e062a38b60876ea85001e36608d8a1c09ef9b098b597dec2df2117087ac611d WatchSource:0}: Error finding container 9e062a38b60876ea85001e36608d8a1c09ef9b098b597dec2df2117087ac611d: Status 404 returned error can't find the container with id 9e062a38b60876ea85001e36608d8a1c09ef9b098b597dec2df2117087ac611d Sep 18 20:01:24.278886 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:24.278867 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.280068 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.280049 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:01:24.310248 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:24.310218 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5f3f19b1eab7558136f3b3fab9b6d0.slice/crio-7e7419453a5108a49cd649ae546b73151723a276be8969e092eda99f82c14704 WatchSource:0}: Error finding container 7e7419453a5108a49cd649ae546b73151723a276be8969e092eda99f82c14704: Status 404 returned error can't find the container with id 7e7419453a5108a49cd649ae546b73151723a276be8969e092eda99f82c14704 Sep 18 20:01:24.379632 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:24.379592 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.480136 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:24.480061 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.580975 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:24.580935 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-232.ec2.internal\" not found" Sep 18 20:01:24.656495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.656469 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 20:01:24.666428 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.666405 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" Sep 18 20:01:24.677198 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.677177 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 20:01:24.678296 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.678263 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" Sep 18 20:01:24.686214 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:24.686191 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 20:01:25.045551 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.045516 2559 apiserver.go:52] "Watching apiserver" Sep 18 20:01:25.053661 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.053627 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 20:01:25.054308 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.054161 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-vjmvb","openshift-network-operator/iptables-alerter-glkb8","openshift-ovn-kubernetes/ovnkube-node-qztvj","kube-system/global-pull-secret-syncer-8fq8w","kube-system/konnectivity-agent-xlkx8","kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal","openshift-cluster-node-tuning-operator/tuned-frxr8","openshift-dns/node-resolver-tgcsb","openshift-image-registry/node-ca-4t6zl","openshift-multus/network-metrics-daemon-7m46g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal","openshift-multus/multus-65zh9","openshift-multus/multus-additional-cni-plugins-lgqn7"] Sep 18 20:01:25.055724 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.055699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.056790 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.056755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.058068 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.058001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.058725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.058447 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.058725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.058633 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.058873 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.058852 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vdtd8\"" Sep 18 20:01:25.059038 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.059017 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.059378 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.059360 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 20:01:25.059468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.059448 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.059527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.059360 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 20:01:25.059685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.059666 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-czwrm\"" Sep 18 20:01:25.059780 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.059725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.059875 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.059843 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:25.061477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.060842 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.061477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.060967 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.061477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.061134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 20:01:25.061477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.061366 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 20:01:25.061477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.061470 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 20:01:25.061752 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.061495 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wmmll\"" Sep 18 20:01:25.062081 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.061954 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 20:01:25.062081 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.062006 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.062246 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.062229 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.063244 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.063225 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 20:01:25.063526 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.063503 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 20:01:25.063610 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.063555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.063854 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.063835 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbs5l\"" Sep 18 20:01:25.064629 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.064612 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 20:01:25.065294 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.065261 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:25.065376 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.065299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfm8w\"" Sep 18 20:01:25.065376 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.065342 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:25.065578 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.065562 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 20:01:25.066102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.066082 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.066364 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.066336 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.069987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.069551 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2jnd\"" Sep 18 20:01:25.069987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.069631 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.069987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.069642 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 20:01:25.069987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.069944 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 20:01:25.072132 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.072110 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkq5k\"" Sep 18 20:01:25.072450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.072427 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 20:01:25.072450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.072442 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.072594 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.072431 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.072594 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.072530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.072689 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.072599 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:25.074042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.074026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.075354 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.075340 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.076693 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.076676 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 20:01:25.076950 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.076933 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.077137 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.077119 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.077632 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.077616 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vp97k\"" Sep 18 20:01:25.078022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.078004 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 20:01:25.078217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.078203 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c52tj\"" Sep 18 20:01:25.078306 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.078225 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 20:01:25.079076 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079022 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8bad512-0d51-4c8d-a288-bcc8c17049e8-kubelet-config\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.079076 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e878eff-e7a1-43a2-bf2c-660de829e65f-konnectivity-ca\") pod \"konnectivity-agent-xlkx8\" (UID: \"2e878eff-e7a1-43a2-bf2c-660de829e65f\") " pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079104 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-cnibin\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-kubelet\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079149 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcxg\" (UniqueName: \"kubernetes.io/projected/d414546a-d81e-4267-9179-a3f81929420b-kube-api-access-wbcxg\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-system-cni-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/030ae6db-d323-4dcc-8242-6750bb91b611-serviceca\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.079256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-var-lib-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079290 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-ovn\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-cnibin\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079373 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-sys\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-tmp\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gd6\" (UniqueName: \"kubernetes.io/projected/2c16f313-cb28-4202-a0a9-3eef01008a52-kube-api-access-65gd6\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e878eff-e7a1-43a2-bf2c-660de829e65f-agent-certs\") pod \"konnectivity-agent-xlkx8\" (UID: \"2e878eff-e7a1-43a2-bf2c-660de829e65f\") " pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-kubernetes\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-tuned\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c16f313-cb28-4202-a0a9-3eef01008a52-ovn-node-metrics-cert\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.079647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079618 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqzh\" (UniqueName: \"kubernetes.io/projected/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-kube-api-access-zxqzh\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-ovnkube-config\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-ovnkube-script-lib\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-conf-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-daemon-config\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-run\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-systemd\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-netns\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.079975 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-host\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-run-netns\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-cni-binary-copy\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-lib-modules\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8bad512-0d51-4c8d-a288-bcc8c17049e8-dbus\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-system-cni-dir\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.080263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-cni-bin\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080295 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-systemd\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-systemd-units\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-log-socket\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-cni-netd\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080405 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb820579-3d7b-4a5b-984d-42b5760d70c8-iptables-alerter-script\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb820579-3d7b-4a5b-984d-42b5760d70c8-host-slash\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-cni-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysctl-d\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-slash\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-etc-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hl8\" (UniqueName: \"kubernetes.io/projected/fb820579-3d7b-4a5b-984d-42b5760d70c8-kube-api-access-f2hl8\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-etc-kubernetes\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-modprobe-d\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysconfig\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.081042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-socket-dir-parent\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030ae6db-d323-4dcc-8242-6750bb91b611-host\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ckq\" (UniqueName: \"kubernetes.io/projected/030ae6db-d323-4dcc-8242-6750bb91b611-kube-api-access-75ckq\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080826 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-kubelet\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55r56\" (UniqueName: \"kubernetes.io/projected/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-kube-api-access-55r56\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080909 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-cni-bin\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysctl-conf\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.080990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-node-log\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-k8s-cni-cncf-io\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081060 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-cni-multus\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-hostroot\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081104 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-multus-certs\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081128 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-var-lib-kubelet\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.081662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrc4\" (UniqueName: \"kubernetes.io/projected/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-kube-api-access-qgrc4\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.082461 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-run-ovn-kubernetes\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.082461 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.081205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-env-overrides\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.104871 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.104845 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7s6n2" Sep 18 20:01:25.116606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.116579 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7s6n2" Sep 18 20:01:25.167656 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.167624 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysctl-conf\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-node-log\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-k8s-cni-cncf-io\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-cni-multus\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181571 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-cni-multus\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-hostroot\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-multus-certs\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-var-lib-kubelet\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrc4\" (UniqueName: \"kubernetes.io/projected/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-kube-api-access-qgrc4\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysctl-conf\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-run-ovn-kubernetes\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-env-overrides\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-k8s-cni-cncf-io\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8bad512-0d51-4c8d-a288-bcc8c17049e8-kubelet-config\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-node-log\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.181831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e878eff-e7a1-43a2-bf2c-660de829e65f-konnectivity-ca\") pod \"konnectivity-agent-xlkx8\" (UID: \"2e878eff-e7a1-43a2-bf2c-660de829e65f\") " pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-var-lib-kubelet\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181848 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-cnibin\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-kubelet\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181904 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcxg\" (UniqueName: \"kubernetes.io/projected/d414546a-d81e-4267-9179-a3f81929420b-kube-api-access-wbcxg\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-hostroot\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-system-cni-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/030ae6db-d323-4dcc-8242-6750bb91b611-serviceca\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-var-lib-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181986 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-kubelet\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-ovn\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-cnibin\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-cnibin\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-cnibin\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-sys\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-tmp\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gd6\" (UniqueName: \"kubernetes.io/projected/2c16f313-cb28-4202-a0a9-3eef01008a52-kube-api-access-65gd6\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.182886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182157 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e878eff-e7a1-43a2-bf2c-660de829e65f-agent-certs\") pod \"konnectivity-agent-xlkx8\" (UID: \"2e878eff-e7a1-43a2-bf2c-660de829e65f\") " pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-kubernetes\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-tuned\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c16f313-cb28-4202-a0a9-3eef01008a52-ovn-node-metrics-cert\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-registration-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182322 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xr5\" (UniqueName: \"kubernetes.io/projected/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-kube-api-access-g6xr5\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e878eff-e7a1-43a2-bf2c-660de829e65f-konnectivity-ca\") pod \"konnectivity-agent-xlkx8\" (UID: \"2e878eff-e7a1-43a2-bf2c-660de829e65f\") " pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182366 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-env-overrides\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-var-lib-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8bad512-0d51-4c8d-a288-bcc8c17049e8-kubelet-config\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.181953 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-multus-certs\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182551 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/030ae6db-d323-4dcc-8242-6750bb91b611-serviceca\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-ovn\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-sys\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" event={"ID":"0a5f3f19b1eab7558136f3b3fab9b6d0","Type":"ContainerStarted","Data":"7e7419453a5108a49cd649ae546b73151723a276be8969e092eda99f82c14704"} Sep 18 20:01:25.183773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-run-ovn-kubernetes\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-kubernetes\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182990 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-system-cni-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.183124 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.183230 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:25.683207303 +0000 UTC m=+4.733567731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.183529 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.183600 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:25.683560562 +0000 UTC m=+4.733920989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.182373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqzh\" (UniqueName: \"kubernetes.io/projected/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-kube-api-access-zxqzh\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-ovnkube-config\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-ovnkube-script-lib\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-conf-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-daemon-config\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-sys-fs\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-run\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.184640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-systemd\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-netns\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-device-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.183975 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-host\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-run-netns\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-cni-binary-copy\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-lib-modules\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8bad512-0d51-4c8d-a288-bcc8c17049e8-dbus\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-system-cni-dir\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-cni-bin\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-hosts-file\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-systemd\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-systemd-units\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-log-socket\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-cni-netd\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.185503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb820579-3d7b-4a5b-984d-42b5760d70c8-iptables-alerter-script\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb820579-3d7b-4a5b-984d-42b5760d70c8-host-slash\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-cni-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysctl-d\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-slash\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.184669 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-systemd-units\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-ovnkube-script-lib\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-conf-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb820579-3d7b-4a5b-984d-42b5760d70c8-iptables-alerter-script\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185511 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-log-socket\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-cni-netd\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-daemon-config\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-run\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.185997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-systemd\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186042 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-run-netns\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.186560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186083 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-run-netns\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-host\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-cni-dir\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb820579-3d7b-4a5b-984d-42b5760d70c8-host-slash\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-lib-modules\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-cni-bin\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysctl-d\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-systemd\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d414546a-d81e-4267-9179-a3f81929420b-system-cni-dir\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8bad512-0d51-4c8d-a288-bcc8c17049e8-dbus\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-cni-binary-copy\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.186961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c16f313-cb28-4202-a0a9-3eef01008a52-ovnkube-config\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-tuned\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-etc-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hl8\" (UniqueName: \"kubernetes.io/projected/fb820579-3d7b-4a5b-984d-42b5760d70c8-kube-api-access-f2hl8\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.187402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-etc-kubernetes\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-socket-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-etc-selinux\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt7p\" (UniqueName: \"kubernetes.io/projected/1a023012-7b42-4f3e-9ccd-b124d7d642d6-kube-api-access-2tt7p\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-modprobe-d\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysconfig\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-socket-dir-parent\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-tmp-dir\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030ae6db-d323-4dcc-8242-6750bb91b611-host\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187726 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ckq\" (UniqueName: \"kubernetes.io/projected/030ae6db-d323-4dcc-8242-6750bb91b611-kube-api-access-75ckq\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-kubelet\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55r56\" (UniqueName: \"kubernetes.io/projected/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-kube-api-access-55r56\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187881 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-cni-bin\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-etc-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187965 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-cni-bin\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.188335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-sysconfig\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-multus-socket-dir-parent\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.187240 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-host-slash\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188101 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030ae6db-d323-4dcc-8242-6750bb91b611-host\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-etc-kubernetes\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188528 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-etc-modprobe-d\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" event={"ID":"5ffd3a0f2bc2beb85186ad42f65a5756","Type":"ContainerStarted","Data":"9e062a38b60876ea85001e36608d8a1c09ef9b098b597dec2df2117087ac611d"} Sep 18 20:01:25.189190 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.188629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c16f313-cb28-4202-a0a9-3eef01008a52-run-openvswitch\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.190549 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.190524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-host-var-lib-kubelet\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.190733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.190711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d414546a-d81e-4267-9179-a3f81929420b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.191936 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.191916 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:25.192030 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.191941 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:25.192030 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.191939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-tmp\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.192030 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.191955 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:25.192176 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.192048 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:01:25.692031677 +0000 UTC m=+4.742392126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:25.193452 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.193382 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c16f313-cb28-4202-a0a9-3eef01008a52-ovn-node-metrics-cert\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.193672 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.193631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e878eff-e7a1-43a2-bf2c-660de829e65f-agent-certs\") pod \"konnectivity-agent-xlkx8\" (UID: \"2e878eff-e7a1-43a2-bf2c-660de829e65f\") " pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.195070 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.194853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gd6\" (UniqueName: \"kubernetes.io/projected/2c16f313-cb28-4202-a0a9-3eef01008a52-kube-api-access-65gd6\") pod \"ovnkube-node-qztvj\" (UID: \"2c16f313-cb28-4202-a0a9-3eef01008a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.195166 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.195149 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqzh\" (UniqueName: \"kubernetes.io/projected/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-kube-api-access-zxqzh\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.198396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.196674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hl8\" (UniqueName: \"kubernetes.io/projected/fb820579-3d7b-4a5b-984d-42b5760d70c8-kube-api-access-f2hl8\") pod \"iptables-alerter-glkb8\" (UID: \"fb820579-3d7b-4a5b-984d-42b5760d70c8\") " pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.198396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.198344 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ckq\" (UniqueName: \"kubernetes.io/projected/030ae6db-d323-4dcc-8242-6750bb91b611-kube-api-access-75ckq\") pod \"node-ca-4t6zl\" (UID: \"030ae6db-d323-4dcc-8242-6750bb91b611\") " pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.198396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.198354 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrc4\" (UniqueName: \"kubernetes.io/projected/ef524e7f-9539-4ef0-a2c6-2f448ba6d361-kube-api-access-qgrc4\") pod \"tuned-frxr8\" (UID: \"ef524e7f-9539-4ef0-a2c6-2f448ba6d361\") " pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.198692 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.198671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcxg\" (UniqueName: \"kubernetes.io/projected/d414546a-d81e-4267-9179-a3f81929420b-kube-api-access-wbcxg\") pod \"multus-additional-cni-plugins-lgqn7\" (UID: \"d414546a-d81e-4267-9179-a3f81929420b\") " pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.200859 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.200817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55r56\" (UniqueName: \"kubernetes.io/projected/be325712-4ebe-4aba-ba1a-62ffa8b9d0d7-kube-api-access-55r56\") pod \"multus-65zh9\" (UID: \"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7\") " pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.225397 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.225310 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-registration-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xr5\" (UniqueName: \"kubernetes.io/projected/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-kube-api-access-g6xr5\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288443 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-sys-fs\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-device-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-hosts-file\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-socket-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-etc-selinux\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-device-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt7p\" (UniqueName: \"kubernetes.io/projected/1a023012-7b42-4f3e-9ccd-b124d7d642d6-kube-api-access-2tt7p\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-tmp-dir\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.288686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-sys-fs\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.289481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-registration-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.289481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-hosts-file\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.289481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-etc-selinux\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.289481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.288814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a023012-7b42-4f3e-9ccd-b124d7d642d6-socket-dir\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.289481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.289063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-tmp-dir\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.297497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.297469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt7p\" (UniqueName: \"kubernetes.io/projected/1a023012-7b42-4f3e-9ccd-b124d7d642d6-kube-api-access-2tt7p\") pod \"aws-ebs-csi-driver-node-t59kj\" (UID: \"1a023012-7b42-4f3e-9ccd-b124d7d642d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.297667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.297517 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xr5\" (UniqueName: \"kubernetes.io/projected/2568c6a7-7806-4cdf-86cb-989ac8ea8e1b-kube-api-access-g6xr5\") pod \"node-resolver-tgcsb\" (UID: \"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b\") " pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.369905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.369869 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-frxr8" Sep 18 20:01:25.379961 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.379929 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65zh9" Sep 18 20:01:25.391372 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.391346 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:25.397412 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.397357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:25.405868 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.405845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" Sep 18 20:01:25.412631 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.412603 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4t6zl" Sep 18 20:01:25.427287 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.427240 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-glkb8" Sep 18 20:01:25.435005 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.434977 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" Sep 18 20:01:25.441893 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.441871 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgcsb" Sep 18 20:01:25.691724 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.691630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:25.691724 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.691674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:25.691953 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.691797 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:25.691953 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.691880 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:26.691858879 +0000 UTC m=+5.742219310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:25.691953 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.691797 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:25.692108 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.691967 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:26.691939142 +0000 UTC m=+5.742299590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:25.792055 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:25.792011 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:25.792247 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.792198 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:25.792247 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.792226 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:25.792247 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.792239 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:25.792384 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:25.792311 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:01:26.792296332 +0000 UTC m=+5.842656763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:26.119369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.119341 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:56:25 +0000 UTC" deadline="2028-03-05 14:58:06.198608445 +0000 UTC" Sep 18 20:01:26.119369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.119369 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12810h56m40.079242493s" Sep 18 20:01:26.128570 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:26.128536 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a023012_7b42_4f3e_9ccd_b124d7d642d6.slice/crio-a61b20447e022997519899986ef6011ba540e4f9349afd5f39e432d6c51a59bb WatchSource:0}: Error finding container a61b20447e022997519899986ef6011ba540e4f9349afd5f39e432d6c51a59bb: Status 404 returned error can't find the container with id a61b20447e022997519899986ef6011ba540e4f9349afd5f39e432d6c51a59bb Sep 18 20:01:26.136532 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:26.136501 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2568c6a7_7806_4cdf_86cb_989ac8ea8e1b.slice/crio-a7727a20cb1e11834b29dfc4b3704a47367bae3674c1220d5ed12d63f0b13323 WatchSource:0}: Error finding container a7727a20cb1e11834b29dfc4b3704a47367bae3674c1220d5ed12d63f0b13323: Status 404 returned error can't find the container with id a7727a20cb1e11834b29dfc4b3704a47367bae3674c1220d5ed12d63f0b13323 Sep 18 20:01:26.137735 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:26.137417 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd414546a_d81e_4267_9179_a3f81929420b.slice/crio-8c3902a4f2636320bf3392bd908ae2c8ff52019fe67aca9c9f062ef1b5236623 WatchSource:0}: Error finding container 8c3902a4f2636320bf3392bd908ae2c8ff52019fe67aca9c9f062ef1b5236623: Status 404 returned error can't find the container with id 8c3902a4f2636320bf3392bd908ae2c8ff52019fe67aca9c9f062ef1b5236623 Sep 18 20:01:26.140125 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:26.139763 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb820579_3d7b_4a5b_984d_42b5760d70c8.slice/crio-34cca3b3938f9d874ec68e483da55fd9fc7bb1acb3e26965df8a55a9cf0fa3e1 WatchSource:0}: Error finding container 34cca3b3938f9d874ec68e483da55fd9fc7bb1acb3e26965df8a55a9cf0fa3e1: Status 404 returned error can't find the container with id 34cca3b3938f9d874ec68e483da55fd9fc7bb1acb3e26965df8a55a9cf0fa3e1 Sep 18 20:01:26.141570 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:26.141187 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e878eff_e7a1_43a2_bf2c_660de829e65f.slice/crio-e8e9f1d99717b511444cb1e91878987045be5674fbf18c40415a501b9965b91f WatchSource:0}: Error finding container e8e9f1d99717b511444cb1e91878987045be5674fbf18c40415a501b9965b91f: Status 404 returned error can't find the container with id e8e9f1d99717b511444cb1e91878987045be5674fbf18c40415a501b9965b91f Sep 18 20:01:26.141908 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:26.141864 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c16f313_cb28_4202_a0a9_3eef01008a52.slice/crio-707d4243799a1b3f617a07f6552e610add930fbb22ed39839906ec520f84197b WatchSource:0}: Error finding container 707d4243799a1b3f617a07f6552e610add930fbb22ed39839906ec520f84197b: Status 404 returned error can't find the container with id 707d4243799a1b3f617a07f6552e610add930fbb22ed39839906ec520f84197b Sep 18 20:01:26.192428 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.192396 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"707d4243799a1b3f617a07f6552e610add930fbb22ed39839906ec520f84197b"} Sep 18 20:01:26.193744 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.193716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-glkb8" event={"ID":"fb820579-3d7b-4a5b-984d-42b5760d70c8","Type":"ContainerStarted","Data":"34cca3b3938f9d874ec68e483da55fd9fc7bb1acb3e26965df8a55a9cf0fa3e1"} Sep 18 20:01:26.194969 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.194941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xlkx8" event={"ID":"2e878eff-e7a1-43a2-bf2c-660de829e65f","Type":"ContainerStarted","Data":"e8e9f1d99717b511444cb1e91878987045be5674fbf18c40415a501b9965b91f"} Sep 18 20:01:26.196317 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.196289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65zh9" event={"ID":"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7","Type":"ContainerStarted","Data":"a1af8ebbace68bc7858a3c23ebad5e6668f10793a0e9793ef0216440a4e25701"} Sep 18 20:01:26.197654 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.197601 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-frxr8" event={"ID":"ef524e7f-9539-4ef0-a2c6-2f448ba6d361","Type":"ContainerStarted","Data":"c3adb4629baeb73395c6a89b89bb847a7f7058977fca79a4378a58b2a9b059c8"} Sep 18 20:01:26.198884 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.198859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" event={"ID":"1a023012-7b42-4f3e-9ccd-b124d7d642d6","Type":"ContainerStarted","Data":"a61b20447e022997519899986ef6011ba540e4f9349afd5f39e432d6c51a59bb"} Sep 18 20:01:26.199999 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.199978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4t6zl" event={"ID":"030ae6db-d323-4dcc-8242-6750bb91b611","Type":"ContainerStarted","Data":"b974a7ce69096a529bb00a8141bde8ec75a5e273698c485261f66372d759c22f"} Sep 18 20:01:26.201191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.201171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerStarted","Data":"8c3902a4f2636320bf3392bd908ae2c8ff52019fe67aca9c9f062ef1b5236623"} Sep 18 20:01:26.202454 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.202432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgcsb" event={"ID":"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b","Type":"ContainerStarted","Data":"a7727a20cb1e11834b29dfc4b3704a47367bae3674c1220d5ed12d63f0b13323"} Sep 18 20:01:26.525832 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.525795 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 20:01:26.698848 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.698753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:26.698848 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.698809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:26.699202 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.698991 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:26.699202 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.699055 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:28.699036726 +0000 UTC m=+7.749397166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:26.699485 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.699465 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:26.699582 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.699524 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:28.699508102 +0000 UTC m=+7.749868528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:26.802610 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:26.802568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:26.802787 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.802763 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:26.802849 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.802787 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:26.802849 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.802800 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:26.802938 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:26.802859 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:01:28.80283966 +0000 UTC m=+7.853200102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:27.120003 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.119690 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:56:25 +0000 UTC" deadline="2028-05-03 15:07:09.237284798 +0000 UTC" Sep 18 20:01:27.120003 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.119957 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14227h5m42.117333362s" Sep 18 20:01:27.180102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.180068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:27.180269 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:27.180202 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:27.180361 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.180311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:27.180416 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:27.180389 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:27.180467 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.180459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:27.180564 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:27.180540 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:27.224340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.224302 2559 generic.go:356] "Generic (PLEG): container finished" podID="0a5f3f19b1eab7558136f3b3fab9b6d0" containerID="fb8912e6c59962e28985e8b1e24b6c62fce1b448f0c27871877826589bc76a22" exitCode=0 Sep 18 20:01:27.224694 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.224618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" event={"ID":"0a5f3f19b1eab7558136f3b3fab9b6d0","Type":"ContainerDied","Data":"fb8912e6c59962e28985e8b1e24b6c62fce1b448f0c27871877826589bc76a22"} Sep 18 20:01:27.237513 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:27.237480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" event={"ID":"5ffd3a0f2bc2beb85186ad42f65a5756","Type":"ContainerStarted","Data":"a5f11aab2478041ff5c3c5caed865048980488ed8616c09462148dd29d268a03"} Sep 18 20:01:28.262937 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:28.262903 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" event={"ID":"0a5f3f19b1eab7558136f3b3fab9b6d0","Type":"ContainerStarted","Data":"9b465a621673150e3804d237e3bad7563491c411d24e97471697747e4951f411"} Sep 18 20:01:28.281084 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:28.280630 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-232.ec2.internal" podStartSLOduration=4.280606574 podStartE2EDuration="4.280606574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:27.258788504 +0000 UTC m=+6.309148953" watchObservedRunningTime="2026-09-18 20:01:28.280606574 +0000 UTC m=+7.330967023" Sep 18 20:01:28.281084 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:28.281044 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-232.ec2.internal" podStartSLOduration=4.281035926 podStartE2EDuration="4.281035926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:28.280369004 +0000 UTC m=+7.330729453" watchObservedRunningTime="2026-09-18 20:01:28.281035926 +0000 UTC m=+7.331396374" Sep 18 20:01:28.720475 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:28.720374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:28.720475 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:28.720427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:28.720753 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.720596 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:28.720753 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.720658 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:32.720638828 +0000 UTC m=+11.770999260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:28.721071 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.721044 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:28.721152 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.721100 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:32.721085459 +0000 UTC m=+11.771445887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:28.820877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:28.820826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:28.821045 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.820985 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:28.821045 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.821008 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:28.821045 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.821020 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:28.821196 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:28.821080 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:01:32.821060021 +0000 UTC m=+11.871420461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:29.178463 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:29.178381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:29.178631 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:29.178521 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:29.179004 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:29.178982 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:29.179106 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:29.179076 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:29.179163 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:29.179124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:29.179210 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:29.179187 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:31.178506 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:31.178462 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:31.178985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:31.178470 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:31.178985 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:31.178610 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:31.178985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:31.178629 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:31.178985 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:31.178714 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:31.178985 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:31.178781 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:32.750977 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:32.750877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:32.750977 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:32.750931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:32.751551 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.751013 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:32.751551 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.751066 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:32.751551 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.751150 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:40.751068139 +0000 UTC m=+19.801428610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:32.751551 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.751168 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:40.751160412 +0000 UTC m=+19.801520839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:32.852178 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:32.852136 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:32.852366 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.852345 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:32.852423 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.852370 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:32.852423 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.852384 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:32.852541 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:32.852445 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:01:40.852425467 +0000 UTC m=+19.902785911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:33.179030 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:33.178998 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:33.179217 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:33.179105 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:33.179217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:33.179166 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:33.179358 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:33.179300 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:33.179358 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:33.179323 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:33.179462 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:33.179378 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:35.178139 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:35.178101 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:35.178562 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:35.178153 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:35.178562 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:35.178104 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:35.178562 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:35.178240 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:35.178562 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:35.178347 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:35.178562 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:35.178419 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:37.178173 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:37.178137 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:37.178648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:37.178137 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:37.178648 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:37.178295 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:37.178648 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:37.178377 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:37.178648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:37.178148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:37.178648 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:37.178480 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:39.177644 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:39.177607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:39.177644 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:39.177636 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:39.178158 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:39.177607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:39.178158 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:39.177737 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:39.178158 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:39.177833 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:39.178158 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:39.177936 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:40.816664 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:40.816622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:40.817295 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:40.816678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:40.817295 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.816786 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:40.817295 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.816797 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:40.817295 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.816856 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:56.816836929 +0000 UTC m=+35.867197372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:40.817295 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.816871 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:56.816865296 +0000 UTC m=+35.867225723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:40.917403 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:40.917370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:40.917575 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.917495 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:40.917575 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.917512 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:40.917575 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.917521 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:40.917720 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:40.917580 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:01:56.917565862 +0000 UTC m=+35.967926293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:41.177888 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:41.177851 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:41.178048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:41.177900 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:41.178048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:41.177851 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:41.178048 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:41.177970 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:41.178155 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:41.178048 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:41.178155 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:41.178117 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:43.177843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:43.177807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:43.178207 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:43.177903 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:43.178207 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:43.177924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:43.178207 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:43.177947 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:43.178207 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:43.178027 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:43.178207 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:43.178104 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:44.297587 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.297350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4t6zl" event={"ID":"030ae6db-d323-4dcc-8242-6750bb91b611","Type":"ContainerStarted","Data":"33bd8beef9e04ad6d20637ed802cee03b190cd4f0bc19e007d2f0d364dfc1d02"} Sep 18 20:01:44.299011 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.298977 2559 generic.go:356] "Generic (PLEG): container finished" podID="d414546a-d81e-4267-9179-a3f81929420b" containerID="18554e28b3b9aadfa72a85584d4ad37590a102496d279de0425ada9f09a39d24" exitCode=0 Sep 18 20:01:44.299131 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.299050 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerDied","Data":"18554e28b3b9aadfa72a85584d4ad37590a102496d279de0425ada9f09a39d24"} Sep 18 20:01:44.300659 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.300626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgcsb" event={"ID":"2568c6a7-7806-4cdf-86cb-989ac8ea8e1b","Type":"ContainerStarted","Data":"0bea9bbb2f5e84e6712348ec4a095e155db3c3be4a17f595b9cfda85e428911a"} Sep 18 20:01:44.303627 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.303610 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:01:44.303952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.303933 2559 generic.go:356] "Generic (PLEG): container finished" podID="2c16f313-cb28-4202-a0a9-3eef01008a52" containerID="058521bebe834f8f854614c9c17fc899322672c3ba1ec1cd80fb6ceb8bb5c4e2" exitCode=1 Sep 18 20:01:44.304020 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.303990 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"e61b63ceaddaaf000197b8f5d47f97b47520ef7568c9c23f2cd03a41f31cd0d3"} Sep 18 20:01:44.304056 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.304043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"9ed77c146a76f6066f608f73949d6607843ca25745d0c4535afeabfa75719965"} Sep 18 20:01:44.304096 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.304089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"6e1e25c6c74fd462ccf047cacab0bb7b96dca234469fee5a617ec329d9afe757"} Sep 18 20:01:44.304133 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.304118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"06538e85ea5317bbd60e53ec487de400d333e8e5143ebcd92a541db9cc21a7a5"} Sep 18 20:01:44.304174 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.304145 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerDied","Data":"058521bebe834f8f854614c9c17fc899322672c3ba1ec1cd80fb6ceb8bb5c4e2"} Sep 18 20:01:44.304226 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.304175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"7cf10eb766810b9b70a1191b4ad18121e0a5108cbca6cbbd0107c9407e9d98b5"} Sep 18 20:01:44.305389 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.305365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xlkx8" event={"ID":"2e878eff-e7a1-43a2-bf2c-660de829e65f","Type":"ContainerStarted","Data":"5da925c1eb3eba6914ae6b1b415395aa21f629e586231dfdfc80930551632090"} Sep 18 20:01:44.306865 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.306843 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65zh9" event={"ID":"be325712-4ebe-4aba-ba1a-62ffa8b9d0d7","Type":"ContainerStarted","Data":"ab4f1a2bd570122579dda70de41b70b4cffcd45aa1610183775b886afce4461d"} Sep 18 20:01:44.308250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.308231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-frxr8" event={"ID":"ef524e7f-9539-4ef0-a2c6-2f448ba6d361","Type":"ContainerStarted","Data":"99a9a8943fb6c4ff1f94acc42feffe29ba997e78fc4959ed35191fd948ea51c5"} Sep 18 20:01:44.309607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.309587 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" event={"ID":"1a023012-7b42-4f3e-9ccd-b124d7d642d6","Type":"ContainerStarted","Data":"73c1c70b07e2d90e67cb05b914839f534bc396926cc085e0e7e6885d5c597778"} Sep 18 20:01:44.313961 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.313830 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4t6zl" podStartSLOduration=3.950397359 podStartE2EDuration="21.313780125s" totalImagesPullingTime="17.363382766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:44.313192513 +0000 UTC m=+23.363552976" watchObservedRunningTime="2026-09-18 20:01:44.313780125 +0000 UTC m=+23.364140577" Sep 18 20:01:44.349894 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.349848 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-frxr8" podStartSLOduration=3.954451164 podStartE2EDuration="21.349831324s" totalImagesPullingTime="17.39538016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:44.349713481 +0000 UTC m=+23.400073928" watchObservedRunningTime="2026-09-18 20:01:44.349831324 +0000 UTC m=+23.400191773" Sep 18 20:01:44.368927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.368868 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-65zh9" podStartSLOduration=3.790350505 podStartE2EDuration="21.36884737s" totalImagesPullingTime="17.578496865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:44.368839848 +0000 UTC m=+23.419200295" watchObservedRunningTime="2026-09-18 20:01:44.36884737 +0000 UTC m=+23.419207819" Sep 18 20:01:44.385268 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.385212 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tgcsb" podStartSLOduration=3.997620566 podStartE2EDuration="21.385190603s" totalImagesPullingTime="17.387570037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:44.384993419 +0000 UTC m=+23.435353866" watchObservedRunningTime="2026-09-18 20:01:44.385190603 +0000 UTC m=+23.435551052" Sep 18 20:01:44.400321 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:44.400249 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xlkx8" podStartSLOduration=11.493108267 podStartE2EDuration="21.400229522s" totalImagesPullingTime="9.907121255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:44.399669871 +0000 UTC m=+23.450030316" watchObservedRunningTime="2026-09-18 20:01:44.400229522 +0000 UTC m=+23.450589971" Sep 18 20:01:45.178518 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.178336 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:45.178662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.178351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:45.178662 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:45.178594 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:45.178761 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:45.178662 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:45.178761 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.178337 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:45.178761 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:45.178724 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:45.194349 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.194320 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 20:01:45.304547 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.304512 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:45.305138 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.305118 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:45.314056 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.314028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-glkb8" event={"ID":"fb820579-3d7b-4a5b-984d-42b5760d70c8","Type":"ContainerStarted","Data":"22a2cde772fa3349e3c1f0de8db653ad9b7d19437d1fd4d2fa6a3532e226229e"} Sep 18 20:01:45.316066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.316039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" event={"ID":"1a023012-7b42-4f3e-9ccd-b124d7d642d6","Type":"ContainerStarted","Data":"5dc8f4c3f3b0bb73d94d70661b52b75a16b146e8ac282d5c28abfdfe8593c21b"} Sep 18 20:01:45.316975 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.316955 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:45.317065 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.317012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xlkx8" Sep 18 20:01:45.346493 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:45.346441 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-glkb8" podStartSLOduration=4.963988012 podStartE2EDuration="22.346426468s" totalImagesPullingTime="17.382438456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:45.345951872 +0000 UTC m=+24.396312319" watchObservedRunningTime="2026-09-18 20:01:45.346426468 +0000 UTC m=+24.396786915" Sep 18 20:01:46.158360 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:46.158221 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T20:01:45.194342252Z","UUID":"b4352095-e88c-4fb8-ac84-7e0fbf3bb8c2","Handler":null,"Name":"","Endpoint":""} Sep 18 20:01:46.159770 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:46.159749 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 20:01:46.159917 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:46.159781 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 20:01:46.323841 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:46.323769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" event={"ID":"1a023012-7b42-4f3e-9ccd-b124d7d642d6","Type":"ContainerStarted","Data":"9607e1a602eb32cf20dc7071f9c9c83fe0b9252c7d0b70bd200b39c0af65d8c3"} Sep 18 20:01:47.178322 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:47.178289 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:47.178536 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:47.178289 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:47.178536 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:47.178401 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:47.178536 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:47.178505 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:47.178536 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:47.178300 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:47.178751 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:47.178715 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:47.331561 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:47.331537 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:01:47.332037 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:47.332014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"15348b53021c2cd3433035ca0f438b125c08b5e6b66dcebc487b8c00d70a74c7"} Sep 18 20:01:47.350685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:47.350631 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t59kj" podStartSLOduration=4.7073679550000005 podStartE2EDuration="24.350610008s" totalImagesPullingTime="19.643242053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:47.350352778 +0000 UTC m=+26.400713227" watchObservedRunningTime="2026-09-18 20:01:47.350610008 +0000 UTC m=+26.400970457" Sep 18 20:01:49.177917 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.177884 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:49.178386 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.177894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:49.178386 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.177997 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:49.178386 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:49.177995 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:49.178386 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:49.178070 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:49.178386 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:49.178138 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:49.342039 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.341798 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:01:49.342919 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.342885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"637f8a9c981cd078c2b21db1f2dde592669f839a42643ce2bb5e3fceb58a9c3b"} Sep 18 20:01:49.343444 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.343316 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:49.343444 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.343386 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:49.343444 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.343399 2559 scope.go:117] "RemoveContainer" containerID="058521bebe834f8f854614c9c17fc899322672c3ba1ec1cd80fb6ceb8bb5c4e2" Sep 18 20:01:49.365263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:49.364971 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:50.348820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.348790 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:01:50.349683 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.349157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" event={"ID":"2c16f313-cb28-4202-a0a9-3eef01008a52","Type":"ContainerStarted","Data":"774d7b027bd22b23a00acacdee38ce54fa2b255bccbd2f5a4baa63e3b452e8f7"} Sep 18 20:01:50.349683 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.349429 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:50.351094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.351066 2559 generic.go:356] "Generic (PLEG): container finished" podID="d414546a-d81e-4267-9179-a3f81929420b" containerID="a87d1ddabeaeb62d4529d1926c62bb946efa2ece60ade64e652b7c8313053a48" exitCode=0 Sep 18 20:01:50.351211 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.351098 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerDied","Data":"a87d1ddabeaeb62d4529d1926c62bb946efa2ece60ade64e652b7c8313053a48"} Sep 18 20:01:50.365087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.365065 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:01:50.384356 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:50.384314 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" podStartSLOduration=9.954423296 podStartE2EDuration="27.384297727s" totalImagesPullingTime="17.429874431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:50.383807154 +0000 UTC m=+29.434167614" watchObservedRunningTime="2026-09-18 20:01:50.384297727 +0000 UTC m=+29.434658172" Sep 18 20:01:51.177683 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.177489 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:51.177952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.177489 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:51.177952 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:51.177765 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:51.177952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.177538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:51.177952 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:51.177832 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:51.177952 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:51.177918 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:51.267135 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.267062 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fq8w"] Sep 18 20:01:51.270593 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.270564 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vjmvb"] Sep 18 20:01:51.271092 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.271071 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7m46g"] Sep 18 20:01:51.356075 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.356048 2559 generic.go:356] "Generic (PLEG): container finished" podID="d414546a-d81e-4267-9179-a3f81929420b" containerID="6848ac339d995c46a6e5694f54ee74c6b15ef09478919fb6e4f5135189f90a59" exitCode=0 Sep 18 20:01:51.356515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.356113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:51.356515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.356150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerDied","Data":"6848ac339d995c46a6e5694f54ee74c6b15ef09478919fb6e4f5135189f90a59"} Sep 18 20:01:51.356515 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:51.356218 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:51.356515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.356301 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:51.356515 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:51.356407 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:51.356515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:51.356439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:51.356708 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:51.356519 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:52.361010 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:52.360972 2559 generic.go:356] "Generic (PLEG): container finished" podID="d414546a-d81e-4267-9179-a3f81929420b" containerID="f4c016beadd018b0a366f5165716c0c69bdc1ed711fda84821cfeade213a60d7" exitCode=0 Sep 18 20:01:52.361432 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:52.361041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerDied","Data":"f4c016beadd018b0a366f5165716c0c69bdc1ed711fda84821cfeade213a60d7"} Sep 18 20:01:53.178657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:53.178577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:53.178831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:53.178688 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:53.178831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:53.178749 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:53.178831 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:53.178810 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:53.178998 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:53.178853 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:53.178998 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:53.178952 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:55.177884 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:55.177843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:55.178363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:55.177954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:55.178363 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:55.177959 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjmvb" podUID="71fbdebd-1d26-48c1-9657-2d5b7763e77e" Sep 18 20:01:55.178363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:55.177990 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:55.178363 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:55.178091 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fq8w" podUID="c8bad512-0d51-4c8d-a288-bcc8c17049e8" Sep 18 20:01:55.178363 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:55.178200 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:01:56.448820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.448260 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-232.ec2.internal" event="NodeReady" Sep 18 20:01:56.448820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.448451 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 20:01:56.491219 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.491184 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-krmvc"] Sep 18 20:01:56.493736 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.493713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.496824 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.496802 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 20:01:56.497263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.497238 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 20:01:56.497475 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.497448 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dw2ht\"" Sep 18 20:01:56.498327 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.498299 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kdqjn"] Sep 18 20:01:56.500122 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.500106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.503250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.502731 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 20:01:56.503250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.502750 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 20:01:56.503250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.503041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xpvnc\"" Sep 18 20:01:56.503250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.503050 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 20:01:56.504015 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.503996 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-krmvc"] Sep 18 20:01:56.509106 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.509083 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdqjn"] Sep 18 20:01:56.633481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.633445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zqg\" (UniqueName: \"kubernetes.io/projected/bb2e0673-cd29-40b6-b85c-15851a3174e6-kube-api-access-v9zqg\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.633481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.633492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e0673-cd29-40b6-b85c-15851a3174e6-tmp-dir\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.633720 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.633556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2e0673-cd29-40b6-b85c-15851a3174e6-config-volume\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.633720 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.633578 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/574561ec-4096-443d-8095-3f7d94cbd239-cert\") pod \"ingress-canary-kdqjn\" (UID: \"574561ec-4096-443d-8095-3f7d94cbd239\") " pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.633720 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.633599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.633720 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.633615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzjp\" (UniqueName: \"kubernetes.io/projected/574561ec-4096-443d-8095-3f7d94cbd239-kube-api-access-gfzjp\") pod \"ingress-canary-kdqjn\" (UID: \"574561ec-4096-443d-8095-3f7d94cbd239\") " pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.734418 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zqg\" (UniqueName: \"kubernetes.io/projected/bb2e0673-cd29-40b6-b85c-15851a3174e6-kube-api-access-v9zqg\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.734418 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e0673-cd29-40b6-b85c-15851a3174e6-tmp-dir\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.734418 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2e0673-cd29-40b6-b85c-15851a3174e6-config-volume\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.734746 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/574561ec-4096-443d-8095-3f7d94cbd239-cert\") pod \"ingress-canary-kdqjn\" (UID: \"574561ec-4096-443d-8095-3f7d94cbd239\") " pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.734746 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.734746 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzjp\" (UniqueName: \"kubernetes.io/projected/574561ec-4096-443d-8095-3f7d94cbd239-kube-api-access-gfzjp\") pod \"ingress-canary-kdqjn\" (UID: \"574561ec-4096-443d-8095-3f7d94cbd239\") " pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.734746 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.734634 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:01:56.734746 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.734694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:57.234675536 +0000 UTC m=+36.285035984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:01:56.734985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.734813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e0673-cd29-40b6-b85c-15851a3174e6-tmp-dir\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.735216 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.735193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2e0673-cd29-40b6-b85c-15851a3174e6-config-volume\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.739584 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.739442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/574561ec-4096-443d-8095-3f7d94cbd239-cert\") pod \"ingress-canary-kdqjn\" (UID: \"574561ec-4096-443d-8095-3f7d94cbd239\") " pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.745567 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.745541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zqg\" (UniqueName: \"kubernetes.io/projected/bb2e0673-cd29-40b6-b85c-15851a3174e6-kube-api-access-v9zqg\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:56.745681 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.745607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzjp\" (UniqueName: \"kubernetes.io/projected/574561ec-4096-443d-8095-3f7d94cbd239-kube-api-access-gfzjp\") pod \"ingress-canary-kdqjn\" (UID: \"574561ec-4096-443d-8095-3f7d94cbd239\") " pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.814722 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.814675 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdqjn" Sep 18 20:01:56.835451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.835416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:56.835451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.835456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:56.835673 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.835611 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:56.835734 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.835670 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:56.835734 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.835692 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret podName:c8bad512-0d51-4c8d-a288-bcc8c17049e8 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:28.835671439 +0000 UTC m=+67.886031868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret") pod "global-pull-secret-syncer-8fq8w" (UID: "c8bad512-0d51-4c8d-a288-bcc8c17049e8") : object "kube-system"/"original-pull-secret" not registered Sep 18 20:01:56.835837 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.835790 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:28.8357687 +0000 UTC m=+67.886129140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 20:01:56.936193 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:56.936158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:56.936412 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.936377 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 20:01:56.936412 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.936402 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 20:01:56.936412 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.936414 2559 projected.go:196] Error preparing data for projected volume kube-api-access-ghmsj for pod openshift-network-diagnostics/network-check-target-vjmvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:56.936577 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:56.936482 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj podName:71fbdebd-1d26-48c1-9657-2d5b7763e77e nodeName:}" failed. No retries permitted until 2026-09-18 20:02:28.936462792 +0000 UTC m=+67.986823237 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghmsj" (UniqueName: "kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj") pod "network-check-target-vjmvb" (UID: "71fbdebd-1d26-48c1-9657-2d5b7763e77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 20:01:57.177544 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.177506 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:01:57.177544 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.177539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:01:57.177793 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.177506 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:01:57.182105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.181867 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 20:01:57.182105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.181933 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4pmvp\"" Sep 18 20:01:57.182105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.181938 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 20:01:57.182105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.181990 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 20:01:57.182105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.181867 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ms6pd\"" Sep 18 20:01:57.182105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.181866 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 20:01:57.239034 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.238992 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:57.239226 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:57.239156 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:01:57.239313 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:57.239233 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:01:58.239213178 +0000 UTC m=+37.289573622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:01:57.543209 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:57.543145 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdqjn"] Sep 18 20:01:57.546969 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:01:57.546946 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574561ec_4096_443d_8095_3f7d94cbd239.slice/crio-84c7ecd38fd458315b61acf47d02ef44938d09410660de92455ff4d47f68e068 WatchSource:0}: Error finding container 84c7ecd38fd458315b61acf47d02ef44938d09410660de92455ff4d47f68e068: Status 404 returned error can't find the container with id 84c7ecd38fd458315b61acf47d02ef44938d09410660de92455ff4d47f68e068 Sep 18 20:01:58.246551 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:58.246514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:01:58.246708 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:58.246674 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:01:58.246774 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:01:58.246743 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:00.246726734 +0000 UTC m=+39.297087180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:01:58.380447 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:58.380411 2559 generic.go:356] "Generic (PLEG): container finished" podID="d414546a-d81e-4267-9179-a3f81929420b" containerID="71a5b0d68b49975500a7e7e88d7fd2921f9089f049828d0199e5ecd52803a6f7" exitCode=0 Sep 18 20:01:58.380627 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:58.380500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerDied","Data":"71a5b0d68b49975500a7e7e88d7fd2921f9089f049828d0199e5ecd52803a6f7"} Sep 18 20:01:58.381821 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:58.381789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdqjn" event={"ID":"574561ec-4096-443d-8095-3f7d94cbd239","Type":"ContainerStarted","Data":"84c7ecd38fd458315b61acf47d02ef44938d09410660de92455ff4d47f68e068"} Sep 18 20:01:59.388812 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:59.388776 2559 generic.go:356] "Generic (PLEG): container finished" podID="d414546a-d81e-4267-9179-a3f81929420b" containerID="6768a02fb0a9b22d5d90a4dfdd0106e5f47021fa24f163ceca11e48359fef314" exitCode=0 Sep 18 20:01:59.389357 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:01:59.388840 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerDied","Data":"6768a02fb0a9b22d5d90a4dfdd0106e5f47021fa24f163ceca11e48359fef314"} Sep 18 20:02:00.263033 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:00.262766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:02:00.263033 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:00.262929 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:02:00.263033 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:00.263039 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:04.263024421 +0000 UTC m=+43.313384847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:02:00.394904 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:00.394879 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" event={"ID":"d414546a-d81e-4267-9179-a3f81929420b","Type":"ContainerStarted","Data":"4bb98d546e8497b0e4e6425461180a9df92905a3e754af64b3e9b34c730eacc0"} Sep 18 20:02:00.439433 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:00.439378 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lgqn7" podStartSLOduration=8.855111354 podStartE2EDuration="37.439362225s" totalImagesPullingTime="28.584250871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:00.438719913 +0000 UTC m=+39.489080373" watchObservedRunningTime="2026-09-18 20:02:00.439362225 +0000 UTC m=+39.489722673" Sep 18 20:02:01.398820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:01.398780 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdqjn" event={"ID":"574561ec-4096-443d-8095-3f7d94cbd239","Type":"ContainerStarted","Data":"74ea9a84436bc6f1e86fa6476c04c9a17bf4dbab33310c70d7a376ea44a95fa7"} Sep 18 20:02:01.415314 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:01.415237 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kdqjn" podStartSLOduration=2.572618114 podStartE2EDuration="5.415222665s" totalImagesPullingTime="2.842604551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:01.41386873 +0000 UTC m=+40.464229178" watchObservedRunningTime="2026-09-18 20:02:01.415222665 +0000 UTC m=+40.465583110" Sep 18 20:02:04.293755 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:04.293711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:02:04.294138 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:04.293871 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:02:04.294138 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:04.293944 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:12.293923963 +0000 UTC m=+51.344284406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:02:12.352256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:12.352210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:02:12.352731 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:12.352349 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:02:12.352731 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:12.352403 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:28.352390158 +0000 UTC m=+67.402750598 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:02:22.371606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:22.371578 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qztvj" Sep 18 20:02:28.366105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.366062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:02:28.366531 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:28.366235 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:02:28.366531 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:28.366318 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:03:00.366301456 +0000 UTC m=+99.416661882 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:02:28.869263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.869227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:02:28.869434 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.869269 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:02:28.872185 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.872165 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 20:02:28.872521 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.872503 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 20:02:28.879761 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:28.879737 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 20:02:28.879877 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:02:28.879819 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:03:32.879796536 +0000 UTC m=+131.930156969 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : secret "metrics-daemon-secret" not found Sep 18 20:02:28.882928 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.882908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8bad512-0d51-4c8d-a288-bcc8c17049e8-original-pull-secret\") pod \"global-pull-secret-syncer-8fq8w\" (UID: \"c8bad512-0d51-4c8d-a288-bcc8c17049e8\") " pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:02:28.969857 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.969818 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:02:28.972876 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.972851 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 20:02:28.983477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.983456 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 20:02:28.993966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:28.993947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmsj\" (UniqueName: \"kubernetes.io/projected/71fbdebd-1d26-48c1-9657-2d5b7763e77e-kube-api-access-ghmsj\") pod \"network-check-target-vjmvb\" (UID: \"71fbdebd-1d26-48c1-9657-2d5b7763e77e\") " pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:02:29.000951 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.000925 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4pmvp\"" Sep 18 20:02:29.002831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.002813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fq8w" Sep 18 20:02:29.008962 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.008943 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:02:29.289585 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.289512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fq8w"] Sep 18 20:02:29.293090 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.292931 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vjmvb"] Sep 18 20:02:29.293817 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:02:29.293791 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bad512_0d51_4c8d_a288_bcc8c17049e8.slice/crio-d277523a7bc59fa522343ea6058743a84379a6fbedcb175c323e03d005e963d9 WatchSource:0}: Error finding container d277523a7bc59fa522343ea6058743a84379a6fbedcb175c323e03d005e963d9: Status 404 returned error can't find the container with id d277523a7bc59fa522343ea6058743a84379a6fbedcb175c323e03d005e963d9 Sep 18 20:02:29.295376 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:02:29.295354 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fbdebd_1d26_48c1_9657_2d5b7763e77e.slice/crio-fbc27ebc01640685ddb656830694ea1d7fb830c186e7082e726b4d46d5f68d36 WatchSource:0}: Error finding container fbc27ebc01640685ddb656830694ea1d7fb830c186e7082e726b4d46d5f68d36: Status 404 returned error can't find the container with id fbc27ebc01640685ddb656830694ea1d7fb830c186e7082e726b4d46d5f68d36 Sep 18 20:02:29.465975 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.465940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fq8w" event={"ID":"c8bad512-0d51-4c8d-a288-bcc8c17049e8","Type":"ContainerStarted","Data":"d277523a7bc59fa522343ea6058743a84379a6fbedcb175c323e03d005e963d9"} Sep 18 20:02:29.467086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:29.467061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vjmvb" event={"ID":"71fbdebd-1d26-48c1-9657-2d5b7763e77e","Type":"ContainerStarted","Data":"fbc27ebc01640685ddb656830694ea1d7fb830c186e7082e726b4d46d5f68d36"} Sep 18 20:02:34.481325 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:34.481257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vjmvb" event={"ID":"71fbdebd-1d26-48c1-9657-2d5b7763e77e","Type":"ContainerStarted","Data":"cb05c4144d9e786d378502376c0cb9fe21c788dd1608064bace469179597f15c"} Sep 18 20:02:34.481877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:34.481375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:02:34.482655 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:34.482628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fq8w" event={"ID":"c8bad512-0d51-4c8d-a288-bcc8c17049e8","Type":"ContainerStarted","Data":"eb02be6d7b3cf85d30fdae19632f446ac22f2d2d3ac231faccbef26eb78113cf"} Sep 18 20:02:34.498269 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:34.498217 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vjmvb" podStartSLOduration=66.948229475 podStartE2EDuration="1m11.498202349s" totalImagesPullingTime="4.549972874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:34.497715537 +0000 UTC m=+73.548075993" watchObservedRunningTime="2026-09-18 20:02:34.498202349 +0000 UTC m=+73.548562796" Sep 18 20:02:34.531189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:34.531131 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8fq8w" podStartSLOduration=66.97681617 podStartE2EDuration="1m11.531113994s" totalImagesPullingTime="4.554297824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:34.530379199 +0000 UTC m=+73.580739647" watchObservedRunningTime="2026-09-18 20:02:34.531113994 +0000 UTC m=+73.581474441" Sep 18 20:02:42.985164 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.985126 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm"] Sep 18 20:02:42.986953 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.986935 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:42.989567 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.989535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 20:02:42.989701 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.989611 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 20:02:42.990943 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.990924 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c5fqw\"" Sep 18 20:02:42.990943 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.990934 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 18 20:02:42.990943 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.990927 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 20:02:42.994768 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.994748 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6"] Sep 18 20:02:42.996659 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.996635 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:42.997433 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.997413 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm"] Sep 18 20:02:42.999407 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.999384 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 18 20:02:42.999504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.999450 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 18 20:02:42.999733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:42.999717 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 18 20:02:43.000074 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.000056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 18 20:02:43.010666 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.010641 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6"] Sep 18 20:02:43.065705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.065705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1804853-fb74-43dc-9e00-efecc2acd25b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d9748b65b-457qm\" (UID: \"f1804853-fb74-43dc-9e00-efecc2acd25b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.065921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.065921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8vs\" (UniqueName: \"kubernetes.io/projected/f17fd22f-7002-4103-a94d-a72a7f0eb453-kube-api-access-5w8vs\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.065921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755gz\" (UniqueName: \"kubernetes.io/projected/f1804853-fb74-43dc-9e00-efecc2acd25b-kube-api-access-755gz\") pod \"managed-serviceaccount-addon-agent-d9748b65b-457qm\" (UID: \"f1804853-fb74-43dc-9e00-efecc2acd25b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.065921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd22f-7002-4103-a94d-a72a7f0eb453-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.065921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-ca\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.065921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.065884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-hub\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.166757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-ca\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.166757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-hub\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.166986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.166986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1804853-fb74-43dc-9e00-efecc2acd25b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d9748b65b-457qm\" (UID: \"f1804853-fb74-43dc-9e00-efecc2acd25b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.166986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.166986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8vs\" (UniqueName: \"kubernetes.io/projected/f17fd22f-7002-4103-a94d-a72a7f0eb453-kube-api-access-5w8vs\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.166986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755gz\" (UniqueName: \"kubernetes.io/projected/f1804853-fb74-43dc-9e00-efecc2acd25b-kube-api-access-755gz\") pod \"managed-serviceaccount-addon-agent-d9748b65b-457qm\" (UID: \"f1804853-fb74-43dc-9e00-efecc2acd25b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.166986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.166938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd22f-7002-4103-a94d-a72a7f0eb453-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.167687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.167659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd22f-7002-4103-a94d-a72a7f0eb453-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.169435 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.169412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.169546 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.169410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-ca\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.169590 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.169578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.169832 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.169811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f17fd22f-7002-4103-a94d-a72a7f0eb453-hub\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.169880 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.169868 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1804853-fb74-43dc-9e00-efecc2acd25b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d9748b65b-457qm\" (UID: \"f1804853-fb74-43dc-9e00-efecc2acd25b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.175308 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.175258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-755gz\" (UniqueName: \"kubernetes.io/projected/f1804853-fb74-43dc-9e00-efecc2acd25b-kube-api-access-755gz\") pod \"managed-serviceaccount-addon-agent-d9748b65b-457qm\" (UID: \"f1804853-fb74-43dc-9e00-efecc2acd25b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.175437 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.175322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8vs\" (UniqueName: \"kubernetes.io/projected/f17fd22f-7002-4103-a94d-a72a7f0eb453-kube-api-access-5w8vs\") pod \"cluster-proxy-proxy-agent-bddb78f4c-tq2m6\" (UID: \"f17fd22f-7002-4103-a94d-a72a7f0eb453\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.305816 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.305708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" Sep 18 20:02:43.313225 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.313195 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:02:43.573549 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.573455 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm"] Sep 18 20:02:43.578363 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:02:43.578332 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1804853_fb74_43dc_9e00_efecc2acd25b.slice/crio-bdb09551aaf5e540b263c9fbd088c9344c00d9216f59e421688f13f471cfc81c WatchSource:0}: Error finding container bdb09551aaf5e540b263c9fbd088c9344c00d9216f59e421688f13f471cfc81c: Status 404 returned error can't find the container with id bdb09551aaf5e540b263c9fbd088c9344c00d9216f59e421688f13f471cfc81c Sep 18 20:02:43.586554 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:43.586526 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6"] Sep 18 20:02:43.589410 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:02:43.589387 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf17fd22f_7002_4103_a94d_a72a7f0eb453.slice/crio-d542814448983a0481fdc0ca9837c52317bc9c5eb00b429215b5fb4870aeb983 WatchSource:0}: Error finding container d542814448983a0481fdc0ca9837c52317bc9c5eb00b429215b5fb4870aeb983: Status 404 returned error can't find the container with id d542814448983a0481fdc0ca9837c52317bc9c5eb00b429215b5fb4870aeb983 Sep 18 20:02:44.511869 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:44.511831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" event={"ID":"f17fd22f-7002-4103-a94d-a72a7f0eb453","Type":"ContainerStarted","Data":"d542814448983a0481fdc0ca9837c52317bc9c5eb00b429215b5fb4870aeb983"} Sep 18 20:02:44.514525 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:44.514492 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" event={"ID":"f1804853-fb74-43dc-9e00-efecc2acd25b","Type":"ContainerStarted","Data":"bdb09551aaf5e540b263c9fbd088c9344c00d9216f59e421688f13f471cfc81c"} Sep 18 20:02:46.523860 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:46.523803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" event={"ID":"f1804853-fb74-43dc-9e00-efecc2acd25b","Type":"ContainerStarted","Data":"d9800b37c0e75125c785d5c1e8b7e9b7350cbd283d0564b53716ec940ff7b08e"} Sep 18 20:02:46.537854 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:46.537796 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" podStartSLOduration=1.722793876 podStartE2EDuration="4.537777794s" totalImagesPullingTime="2.814983918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:46.5371144 +0000 UTC m=+85.587474851" watchObservedRunningTime="2026-09-18 20:02:46.537777794 +0000 UTC m=+85.588138238" Sep 18 20:02:48.532610 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:48.532570 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" event={"ID":"f17fd22f-7002-4103-a94d-a72a7f0eb453","Type":"ContainerStarted","Data":"8476670c0760206b5f48a833b79295d33bb9ab4b80aa0068c3551a911ed39ad2"} Sep 18 20:02:48.532610 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:48.532616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" event={"ID":"f17fd22f-7002-4103-a94d-a72a7f0eb453","Type":"ContainerStarted","Data":"e018e3a747214575ffbe79826525ab81f73abcb98df80455c26a05056ca8ee3b"} Sep 18 20:02:48.533038 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:48.532629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" event={"ID":"f17fd22f-7002-4103-a94d-a72a7f0eb453","Type":"ContainerStarted","Data":"df26ac5523fe8741c54adee753573ad269f3bdd1126dcd373f801cd3abe1e699"} Sep 18 20:02:48.552222 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:02:48.552169 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" podStartSLOduration=2.544428913 podStartE2EDuration="6.552151719s" totalImagesPullingTime="4.007722806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:48.551395496 +0000 UTC m=+87.601755944" watchObservedRunningTime="2026-09-18 20:02:48.552151719 +0000 UTC m=+87.602512183" Sep 18 20:03:00.392094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:03:00.392052 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:03:00.392591 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:03:00.392206 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:03:00.392591 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:03:00.392300 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:04:04.392268199 +0000 UTC m=+163.442628628 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:03:05.487757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:03:05.487729 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vjmvb" Sep 18 20:03:32.927898 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:03:32.927857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:03:32.928437 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:03:32.928033 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 20:03:32.928437 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:03:32.928119 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs podName:6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472 nodeName:}" failed. No retries permitted until 2026-09-18 20:05:34.928102105 +0000 UTC m=+253.978462530 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs") pod "network-metrics-daemon-7m46g" (UID: "6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472") : secret "metrics-daemon-secret" not found Sep 18 20:03:59.507071 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:03:59.507023 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-krmvc" podUID="bb2e0673-cd29-40b6-b85c-15851a3174e6" Sep 18 20:03:59.735409 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:03:59.735373 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-krmvc" Sep 18 20:04:00.190465 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:04:00.190416 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7m46g" podUID="6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472" Sep 18 20:04:04.445873 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:04:04.445830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:04:04.446299 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:04:04.445961 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 20:04:04.446299 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:04:04.446013 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls podName:bb2e0673-cd29-40b6-b85c-15851a3174e6 nodeName:}" failed. No retries permitted until 2026-09-18 20:06:06.445999438 +0000 UTC m=+285.496359864 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls") pod "dns-default-krmvc" (UID: "bb2e0673-cd29-40b6-b85c-15851a3174e6") : secret "dns-default-metrics-tls" not found Sep 18 20:04:13.178760 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:04:13.178720 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:04:46.875448 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:04:46.875412 2559 generic.go:356] "Generic (PLEG): container finished" podID="f1804853-fb74-43dc-9e00-efecc2acd25b" containerID="d9800b37c0e75125c785d5c1e8b7e9b7350cbd283d0564b53716ec940ff7b08e" exitCode=255 Sep 18 20:04:46.875852 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:04:46.875487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" event={"ID":"f1804853-fb74-43dc-9e00-efecc2acd25b","Type":"ContainerDied","Data":"d9800b37c0e75125c785d5c1e8b7e9b7350cbd283d0564b53716ec940ff7b08e"} Sep 18 20:04:46.875852 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:04:46.875794 2559 scope.go:117] "RemoveContainer" containerID="d9800b37c0e75125c785d5c1e8b7e9b7350cbd283d0564b53716ec940ff7b08e" Sep 18 20:04:47.880231 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:04:47.880194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d9748b65b-457qm" event={"ID":"f1804853-fb74-43dc-9e00-efecc2acd25b","Type":"ContainerStarted","Data":"e51dcc8626a56a75eb6c14f5de6c1fd4bb038b4ce77789ef9b19e706e8193d80"} Sep 18 20:05:07.235586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.235550 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7plkl"] Sep 18 20:05:07.238534 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.238513 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.241111 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.241081 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 20:05:07.241111 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.241081 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 20:05:07.241267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.241086 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 20:05:07.242368 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.242354 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 20:05:07.242437 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.242423 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7xss\"" Sep 18 20:05:07.250354 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.250332 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7plkl"] Sep 18 20:05:07.380522 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.380485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a07ae61c-d277-4b52-8026-9f49b7fb2deb-crio-socket\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.380522 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.380521 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.380780 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.380548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6mq\" (UniqueName: \"kubernetes.io/projected/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-api-access-gg6mq\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.380780 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.380570 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07ae61c-d277-4b52-8026-9f49b7fb2deb-data-volume\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.380780 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.380666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482015 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.481979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482199 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.482022 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6mq\" (UniqueName: \"kubernetes.io/projected/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-api-access-gg6mq\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482199 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.482044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07ae61c-d277-4b52-8026-9f49b7fb2deb-data-volume\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482199 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.482125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482369 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.482198 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 20:05:07.482369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.482262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a07ae61c-d277-4b52-8026-9f49b7fb2deb-crio-socket\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482369 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.482320 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 20:05:07.482369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.482204 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a07ae61c-d277-4b52-8026-9f49b7fb2deb-crio-socket\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482500 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.482320 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:07.982268487 +0000 UTC m=+227.032628915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : configmap "kube-rbac-proxy" not found Sep 18 20:05:07.482500 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.482418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07ae61c-d277-4b52-8026-9f49b7fb2deb-data-volume\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.482500 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.482430 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:07.98240233 +0000 UTC m=+227.032762759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : secret "insights-runtime-extractor-tls" not found Sep 18 20:05:07.490491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.490424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6mq\" (UniqueName: \"kubernetes.io/projected/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-api-access-gg6mq\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.985839 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.985801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.986057 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:07.985863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:07.986057 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.985936 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 20:05:07.986057 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.985975 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 20:05:07.986057 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.985992 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:08.985976523 +0000 UTC m=+228.036336950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : configmap "kube-rbac-proxy" not found Sep 18 20:05:07.986057 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:07.986049 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:08.986033225 +0000 UTC m=+228.036393676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : secret "insights-runtime-extractor-tls" not found Sep 18 20:05:08.993145 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:08.993102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:08.993576 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:08.993163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:08.993576 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:08.993221 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 20:05:08.993576 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:08.993267 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 20:05:08.993576 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:08.993305 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:10.993289893 +0000 UTC m=+230.043650330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : configmap "kube-rbac-proxy" not found Sep 18 20:05:08.993576 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:08.993324 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:10.993313177 +0000 UTC m=+230.043673603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : secret "insights-runtime-extractor-tls" not found Sep 18 20:05:10.217936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:10.217909 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgcsb_2568c6a7-7806-4cdf-86cb-989ac8ea8e1b/dns-node-resolver/0.log" Sep 18 20:05:10.817633 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:10.817605 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4t6zl_030ae6db-d323-4dcc-8242-6750bb91b611/node-ca/0.log" Sep 18 20:05:11.007549 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:11.007514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:11.007744 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:11.007583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:11.007744 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:11.007638 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 20:05:11.007744 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:11.007705 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:15.007690088 +0000 UTC m=+234.058050513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : configmap "kube-rbac-proxy" not found Sep 18 20:05:11.007744 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:11.007719 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 20:05:11.007912 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:11.007771 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:15.007755352 +0000 UTC m=+234.058115785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : secret "insights-runtime-extractor-tls" not found Sep 18 20:05:11.621259 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:11.621226 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdqjn_574561ec-4096-443d-8095-3f7d94cbd239/serve-healthcheck-canary/0.log" Sep 18 20:05:15.038789 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:15.038742 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:15.039197 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:15.038888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:15.039197 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:15.038897 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 20:05:15.039197 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:15.038969 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 20:05:15.039197 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:15.038995 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:23.038977729 +0000 UTC m=+242.089338154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : secret "insights-runtime-extractor-tls" not found Sep 18 20:05:15.039197 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:15.039017 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm podName:a07ae61c-d277-4b52-8026-9f49b7fb2deb nodeName:}" failed. No retries permitted until 2026-09-18 20:05:23.039005242 +0000 UTC m=+242.089365668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7plkl" (UID: "a07ae61c-d277-4b52-8026-9f49b7fb2deb") : configmap "kube-rbac-proxy" not found Sep 18 20:05:23.108440 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.108408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:23.108830 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.108552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:23.109016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.108997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a07ae61c-d277-4b52-8026-9f49b7fb2deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:23.110667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.110643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a07ae61c-d277-4b52-8026-9f49b7fb2deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7plkl\" (UID: \"a07ae61c-d277-4b52-8026-9f49b7fb2deb\") " pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:23.149713 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.149683 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7xss\"" Sep 18 20:05:23.157866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.157825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7plkl" Sep 18 20:05:23.430801 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.430712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7plkl"] Sep 18 20:05:23.433588 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:23.433562 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ae61c_d277_4b52_8026_9f49b7fb2deb.slice/crio-865db9dfaf4c6cc8d009e9fb33181ef91bb513f37c275e801c7643c89805d070 WatchSource:0}: Error finding container 865db9dfaf4c6cc8d009e9fb33181ef91bb513f37c275e801c7643c89805d070: Status 404 returned error can't find the container with id 865db9dfaf4c6cc8d009e9fb33181ef91bb513f37c275e801c7643c89805d070 Sep 18 20:05:23.986970 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.986934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7plkl" event={"ID":"a07ae61c-d277-4b52-8026-9f49b7fb2deb","Type":"ContainerStarted","Data":"326c0d810b338581f2fd65686c8895d4ef6897451d62a7f828ac91090bfdddf1"} Sep 18 20:05:23.987151 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:23.986987 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7plkl" event={"ID":"a07ae61c-d277-4b52-8026-9f49b7fb2deb","Type":"ContainerStarted","Data":"865db9dfaf4c6cc8d009e9fb33181ef91bb513f37c275e801c7643c89805d070"} Sep 18 20:05:24.991891 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:24.991854 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7plkl" event={"ID":"a07ae61c-d277-4b52-8026-9f49b7fb2deb","Type":"ContainerStarted","Data":"3423620a0a172dcbb283d230db229e56e2e6e0ea093f3b6ce8311cd129486070"} Sep 18 20:05:25.996544 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:25.996451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7plkl" event={"ID":"a07ae61c-d277-4b52-8026-9f49b7fb2deb","Type":"ContainerStarted","Data":"43265fc48105a942dacb94035863bc8c76465eb1a07bb1fce25fc799a126870a"} Sep 18 20:05:26.012740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:26.012688 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7plkl" podStartSLOduration=16.921611946 podStartE2EDuration="19.012672498s" totalImagesPullingTime="2.091060552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:05:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:26.012463859 +0000 UTC m=+245.062824308" watchObservedRunningTime="2026-09-18 20:05:26.012672498 +0000 UTC m=+245.063032947" Sep 18 20:05:34.992336 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:34.992233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:05:34.994546 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:34.994523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472-metrics-certs\") pod \"network-metrics-daemon-7m46g\" (UID: \"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472\") " pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:05:35.082237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:35.082203 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ms6pd\"" Sep 18 20:05:35.089828 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:35.089808 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m46g" Sep 18 20:05:35.359364 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:35.359330 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7m46g"] Sep 18 20:05:35.362361 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:35.362323 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a32d2ac_9ee8_4f19_8fc3_3b18d4a29472.slice/crio-ac52b98ba4725a458fc20a27393ab83f1401a4fd058482fd461a9251b5201ba0 WatchSource:0}: Error finding container ac52b98ba4725a458fc20a27393ab83f1401a4fd058482fd461a9251b5201ba0: Status 404 returned error can't find the container with id ac52b98ba4725a458fc20a27393ab83f1401a4fd058482fd461a9251b5201ba0 Sep 18 20:05:36.032189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:36.032153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7m46g" event={"ID":"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472","Type":"ContainerStarted","Data":"ac52b98ba4725a458fc20a27393ab83f1401a4fd058482fd461a9251b5201ba0"} Sep 18 20:05:37.037513 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.037474 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7m46g" event={"ID":"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472","Type":"ContainerStarted","Data":"952a62f00ac5507af1cdb65737dccc0e6333c2de87a2ccd80f2931d62cac0f94"} Sep 18 20:05:37.037886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.037531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7m46g" event={"ID":"6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472","Type":"ContainerStarted","Data":"c17cd742ad729473a80d8b83197bc6cc2d3c6d0466bcf1fc7a7580ebd2529b93"} Sep 18 20:05:37.053531 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.053473 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7m46g" podStartSLOduration=253.181871196 podStartE2EDuration="4m14.053456292s" totalImagesPullingTime="871.585096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:37.052325502 +0000 UTC m=+256.102685951" watchObservedRunningTime="2026-09-18 20:05:37.053456292 +0000 UTC m=+256.103816718" Sep 18 20:05:37.697858 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.697820 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7"] Sep 18 20:05:37.700687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.700669 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:37.703437 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.703411 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x9mcj\"" Sep 18 20:05:37.703674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.703650 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 18 20:05:37.711905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.711880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7"] Sep 18 20:05:37.713289 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.713253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e7deb29-dc62-4291-b7e7-1c92b0a3e48a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w6gr7\" (UID: \"2e7deb29-dc62-4291-b7e7-1c92b0a3e48a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:37.813931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:37.813898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e7deb29-dc62-4291-b7e7-1c92b0a3e48a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w6gr7\" (UID: \"2e7deb29-dc62-4291-b7e7-1c92b0a3e48a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:37.814112 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:37.814061 2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 18 20:05:37.814164 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:05:37.814131 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7deb29-dc62-4291-b7e7-1c92b0a3e48a-tls-certificates podName:2e7deb29-dc62-4291-b7e7-1c92b0a3e48a nodeName:}" failed. No retries permitted until 2026-09-18 20:05:38.314112251 +0000 UTC m=+257.364472677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2e7deb29-dc62-4291-b7e7-1c92b0a3e48a-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" (UID: "2e7deb29-dc62-4291-b7e7-1c92b0a3e48a") : secret "prometheus-operator-admission-webhook-tls" not found Sep 18 20:05:38.317739 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:38.317699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e7deb29-dc62-4291-b7e7-1c92b0a3e48a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w6gr7\" (UID: \"2e7deb29-dc62-4291-b7e7-1c92b0a3e48a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:38.320355 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:38.320315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e7deb29-dc62-4291-b7e7-1c92b0a3e48a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w6gr7\" (UID: \"2e7deb29-dc62-4291-b7e7-1c92b0a3e48a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:38.609186 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:38.609150 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:38.880768 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:38.880687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7"] Sep 18 20:05:38.884739 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:38.884713 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7deb29_dc62_4291_b7e7_1c92b0a3e48a.slice/crio-0f87bf3e3e6f1e3c9113751a6029f5cbf85407657a7519ba2202fa33afde9a18 WatchSource:0}: Error finding container 0f87bf3e3e6f1e3c9113751a6029f5cbf85407657a7519ba2202fa33afde9a18: Status 404 returned error can't find the container with id 0f87bf3e3e6f1e3c9113751a6029f5cbf85407657a7519ba2202fa33afde9a18 Sep 18 20:05:39.050248 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.050206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" event={"ID":"2e7deb29-dc62-4291-b7e7-1c92b0a3e48a","Type":"ContainerStarted","Data":"0f87bf3e3e6f1e3c9113751a6029f5cbf85407657a7519ba2202fa33afde9a18"} Sep 18 20:05:39.840892 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.840860 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bd4dddcf-56zx6"] Sep 18 20:05:39.843645 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.843628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.846146 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.846125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 20:05:39.847343 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847323 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 20:05:39.847449 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847347 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 20:05:39.847449 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847327 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j65gf\"" Sep 18 20:05:39.847449 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847440 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 20:05:39.847607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847468 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 20:05:39.847607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847501 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 20:05:39.847607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.847516 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 20:05:39.852261 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.852239 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 18 20:05:39.853579 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.853522 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bd4dddcf-56zx6"] Sep 18 20:05:39.929254 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-console-config\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.929460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-serving-cert\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.929460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-trusted-ca-bundle\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.929460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-service-ca\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.929586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-oauth-serving-cert\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.929586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-oauth-config\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:39.929586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:39.929568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwgz\" (UniqueName: \"kubernetes.io/projected/567e59db-5c42-4881-b981-6c2fcfbb7198-kube-api-access-lzwgz\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030347 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-service-ca\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030347 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-oauth-serving-cert\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030603 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-oauth-config\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030603 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwgz\" (UniqueName: \"kubernetes.io/projected/567e59db-5c42-4881-b981-6c2fcfbb7198-kube-api-access-lzwgz\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030603 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-console-config\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030603 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-serving-cert\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.030603 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.030525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-trusted-ca-bundle\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.031218 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.031166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-service-ca\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.031360 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.031267 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-console-config\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.031525 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.031498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-trusted-ca-bundle\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.031751 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.031714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-oauth-serving-cert\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.033324 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.033302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-oauth-config\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.033426 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.033302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-serving-cert\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.038344 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.038317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwgz\" (UniqueName: \"kubernetes.io/projected/567e59db-5c42-4881-b981-6c2fcfbb7198-kube-api-access-lzwgz\") pod \"console-9bd4dddcf-56zx6\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.155155 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.155111 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:40.467386 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:40.467310 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bd4dddcf-56zx6"] Sep 18 20:05:40.470190 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:40.470159 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e59db_5c42_4881_b981_6c2fcfbb7198.slice/crio-47afa924ccfe8dfe941dc0fb958b9cabffccbcbbced3ed53f8e701ceec160fdd WatchSource:0}: Error finding container 47afa924ccfe8dfe941dc0fb958b9cabffccbcbbced3ed53f8e701ceec160fdd: Status 404 returned error can't find the container with id 47afa924ccfe8dfe941dc0fb958b9cabffccbcbbced3ed53f8e701ceec160fdd Sep 18 20:05:41.061916 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:41.061842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" event={"ID":"2e7deb29-dc62-4291-b7e7-1c92b0a3e48a","Type":"ContainerStarted","Data":"f3352ae30c3a61b2c0a4af0a3e39bf11f56624bcf50f0c3275ba9872c9de116c"} Sep 18 20:05:41.062455 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:41.062339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:41.066321 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:41.065191 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bd4dddcf-56zx6" event={"ID":"567e59db-5c42-4881-b981-6c2fcfbb7198","Type":"ContainerStarted","Data":"47afa924ccfe8dfe941dc0fb958b9cabffccbcbbced3ed53f8e701ceec160fdd"} Sep 18 20:05:41.069951 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:41.069896 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" Sep 18 20:05:41.078798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:41.078629 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w6gr7" podStartSLOduration=2.718524674 podStartE2EDuration="4.078606038s" totalImagesPullingTime="1.360081364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:41.077227162 +0000 UTC m=+260.127587611" watchObservedRunningTime="2026-09-18 20:05:41.078606038 +0000 UTC m=+260.128966488" Sep 18 20:05:45.083353 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:45.083316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bd4dddcf-56zx6" event={"ID":"567e59db-5c42-4881-b981-6c2fcfbb7198","Type":"ContainerStarted","Data":"083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2"} Sep 18 20:05:45.101076 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:45.101020 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9bd4dddcf-56zx6" podStartSLOduration=2.17285543 podStartE2EDuration="6.101003457s" totalImagesPullingTime="3.928148027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:45.099825162 +0000 UTC m=+264.150185610" watchObservedRunningTime="2026-09-18 20:05:45.101003457 +0000 UTC m=+264.151363902" Sep 18 20:05:47.182498 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.182470 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2wgnp"] Sep 18 20:05:47.185764 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.185745 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.188930 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.188894 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 20:05:47.189763 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.189737 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 20:05:47.189763 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.189756 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 20:05:47.189941 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.189827 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 20:05:47.189941 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.189849 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvmbl\"" Sep 18 20:05:47.189941 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.189860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 20:05:47.189941 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.189849 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 20:05:47.280995 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.280955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-sys\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-wtmp\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281057 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2f6j\" (UniqueName: \"kubernetes.io/projected/f3debf0f-62b6-44b4-824b-506fab50d6a4-kube-api-access-t2f6j\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-textfile\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-root\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281431 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281212 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-tls\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281431 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.281431 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.281304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3debf0f-62b6-44b4-824b-506fab50d6a4-metrics-client-ca\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382089 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382089 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-wtmp\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2f6j\" (UniqueName: \"kubernetes.io/projected/f3debf0f-62b6-44b4-824b-506fab50d6a4-kube-api-access-t2f6j\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-textfile\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-root\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-tls\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3debf0f-62b6-44b4-824b-506fab50d6a4-metrics-client-ca\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-sys\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-root\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382289 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-wtmp\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.382363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3debf0f-62b6-44b4-824b-506fab50d6a4-sys\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.383027 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.382967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-textfile\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.383245 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.383221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3debf0f-62b6-44b4-824b-506fab50d6a4-metrics-client-ca\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.383415 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.383390 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.388397 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.388370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.388589 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.388570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3debf0f-62b6-44b4-824b-506fab50d6a4-node-exporter-tls\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.390157 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.390139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2f6j\" (UniqueName: \"kubernetes.io/projected/f3debf0f-62b6-44b4-824b-506fab50d6a4-kube-api-access-t2f6j\") pod \"node-exporter-2wgnp\" (UID: \"f3debf0f-62b6-44b4-824b-506fab50d6a4\") " pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.494607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:47.494518 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wgnp" Sep 18 20:05:47.506048 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:47.506011 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3debf0f_62b6_44b4_824b_506fab50d6a4.slice/crio-eabea125a500f8b551465ca1e2516c872c104fcf6d683b91af122b0c23d9a65a WatchSource:0}: Error finding container eabea125a500f8b551465ca1e2516c872c104fcf6d683b91af122b0c23d9a65a: Status 404 returned error can't find the container with id eabea125a500f8b551465ca1e2516c872c104fcf6d683b91af122b0c23d9a65a Sep 18 20:05:48.095263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.095217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wgnp" event={"ID":"f3debf0f-62b6-44b4-824b-506fab50d6a4","Type":"ContainerStarted","Data":"eabea125a500f8b551465ca1e2516c872c104fcf6d683b91af122b0c23d9a65a"} Sep 18 20:05:48.197927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.197896 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 20:05:48.201246 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.201231 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.204043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204016 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 18 20:05:48.204152 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204061 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 18 20:05:48.204476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204354 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 18 20:05:48.204476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204371 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 18 20:05:48.204476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204404 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 18 20:05:48.204476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204404 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 18 20:05:48.204476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204471 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l4m78\"" Sep 18 20:05:48.204785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204489 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 18 20:05:48.204785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204510 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 18 20:05:48.205004 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.204980 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 18 20:05:48.216142 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.216119 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 20:05:48.288176 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288373 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288212 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288373 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqcc\" (UniqueName: \"kubernetes.io/projected/c7194915-7361-4d15-8e63-4b9243aaa185-kube-api-access-ptqcc\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288373 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288373 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288338 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-web-config\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7194915-7361-4d15-8e63-4b9243aaa185-config-out\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7194915-7361-4d15-8e63-4b9243aaa185-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7194915-7361-4d15-8e63-4b9243aaa185-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7194915-7361-4d15-8e63-4b9243aaa185-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288549 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-config-volume\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.288648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.288610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c7194915-7361-4d15-8e63-4b9243aaa185-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.389993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.389955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.389993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.389994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqcc\" (UniqueName: \"kubernetes.io/projected/c7194915-7361-4d15-8e63-4b9243aaa185-kube-api-access-ptqcc\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390023 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-web-config\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7194915-7361-4d15-8e63-4b9243aaa185-config-out\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7194915-7361-4d15-8e63-4b9243aaa185-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7194915-7361-4d15-8e63-4b9243aaa185-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7194915-7361-4d15-8e63-4b9243aaa185-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390661 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-config-volume\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390661 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c7194915-7361-4d15-8e63-4b9243aaa185-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.390661 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.390318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.391737 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.391704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7194915-7361-4d15-8e63-4b9243aaa185-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.392014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.391985 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c7194915-7361-4d15-8e63-4b9243aaa185-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.392099 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.391987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7194915-7361-4d15-8e63-4b9243aaa185-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.394316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.394170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7194915-7361-4d15-8e63-4b9243aaa185-config-out\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.394425 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.394374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-web-config\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.394508 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.394483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.394871 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.394854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7194915-7361-4d15-8e63-4b9243aaa185-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.394962 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.394903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.394962 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.394918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-config-volume\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.395288 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.395256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.395679 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.395657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.395951 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.395934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7194915-7361-4d15-8e63-4b9243aaa185-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.397768 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.397747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqcc\" (UniqueName: \"kubernetes.io/projected/c7194915-7361-4d15-8e63-4b9243aaa185-kube-api-access-ptqcc\") pod \"alertmanager-main-0\" (UID: \"c7194915-7361-4d15-8e63-4b9243aaa185\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.510759 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.510718 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 20:05:48.863815 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:48.863782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 20:05:48.867351 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:48.867324 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7194915_7361_4d15_8e63_4b9243aaa185.slice/crio-ab626118fbd07359d7a82ed9e3a64452d2d6329b82c646d244128afb8db41809 WatchSource:0}: Error finding container ab626118fbd07359d7a82ed9e3a64452d2d6329b82c646d244128afb8db41809: Status 404 returned error can't find the container with id ab626118fbd07359d7a82ed9e3a64452d2d6329b82c646d244128afb8db41809 Sep 18 20:05:49.099678 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:49.099650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"ab626118fbd07359d7a82ed9e3a64452d2d6329b82c646d244128afb8db41809"} Sep 18 20:05:49.101122 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:49.101098 2559 generic.go:356] "Generic (PLEG): container finished" podID="f3debf0f-62b6-44b4-824b-506fab50d6a4" containerID="af8067c56558377584c2d69009b943ccc8bfaf35a54dc07757060bfdbd62381c" exitCode=0 Sep 18 20:05:49.101227 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:49.101136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wgnp" event={"ID":"f3debf0f-62b6-44b4-824b-506fab50d6a4","Type":"ContainerDied","Data":"af8067c56558377584c2d69009b943ccc8bfaf35a54dc07757060bfdbd62381c"} Sep 18 20:05:50.106104 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.106072 2559 generic.go:356] "Generic (PLEG): container finished" podID="c7194915-7361-4d15-8e63-4b9243aaa185" containerID="6b0ef9f5a155a3150f7a21e8c1242358aeb6466a7326e3d102d98a9ca8bb7bb8" exitCode=0 Sep 18 20:05:50.106542 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.106148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerDied","Data":"6b0ef9f5a155a3150f7a21e8c1242358aeb6466a7326e3d102d98a9ca8bb7bb8"} Sep 18 20:05:50.108488 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.108464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wgnp" event={"ID":"f3debf0f-62b6-44b4-824b-506fab50d6a4","Type":"ContainerStarted","Data":"07f0bc05b55aa552d0252845d88811d996d31a39c7f493a3fbd6fea300e4ddcc"} Sep 18 20:05:50.108583 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.108511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wgnp" event={"ID":"f3debf0f-62b6-44b4-824b-506fab50d6a4","Type":"ContainerStarted","Data":"45cb9afdd6a6a4bbdbe53da95c166f0de3124e891d0f4bc98de9f3209b779f52"} Sep 18 20:05:50.156105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.156066 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:50.156300 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.156160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:05:50.157502 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.157474 2559 patch_prober.go:28] interesting pod/console-9bd4dddcf-56zx6 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.12:8443/health\": dial tcp 10.134.0.12:8443: connect: connection refused" start-of-body= Sep 18 20:05:50.157617 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.157533 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-9bd4dddcf-56zx6" podUID="567e59db-5c42-4881-b981-6c2fcfbb7198" containerName="console" probeResult="failure" output="Get \"https://10.134.0.12:8443/health\": dial tcp 10.134.0.12:8443: connect: connection refused" Sep 18 20:05:50.167201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.167149 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2wgnp" podStartSLOduration=1.971465835 podStartE2EDuration="3.167133915s" totalImagesPullingTime="1.134088677s" totalInitContainerRuntime="61.579403ms" isStatefulPod=true podCreationTimestamp="2026-09-18 20:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:50.166332497 +0000 UTC m=+269.216692950" watchObservedRunningTime="2026-09-18 20:05:50.167133915 +0000 UTC m=+269.217494361" Sep 18 20:05:50.215974 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.215925 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv"] Sep 18 20:05:50.219430 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.219412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.222127 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222098 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1g8kovljtrkue\"" Sep 18 20:05:50.222127 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222116 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 18 20:05:50.222533 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222153 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 18 20:05:50.222533 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222157 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4wsjd\"" Sep 18 20:05:50.222533 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222410 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 18 20:05:50.222533 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222492 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 18 20:05:50.222676 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.222668 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 18 20:05:50.234707 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.234680 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv"] Sep 18 20:05:50.306252 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306252 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbf0fba-4548-49d5-b602-1522ac7872fb-metrics-client-ca\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306252 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-grpc-tls\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306327 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306401 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jz9l\" (UniqueName: \"kubernetes.io/projected/ddbf0fba-4548-49d5-b602-1522ac7872fb-kube-api-access-6jz9l\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.306491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.306476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-tls\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407186 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407186 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407188 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbf0fba-4548-49d5-b602-1522ac7872fb-metrics-client-ca\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-grpc-tls\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407476 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz9l\" (UniqueName: \"kubernetes.io/projected/ddbf0fba-4548-49d5-b602-1522ac7872fb-kube-api-access-6jz9l\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.407724 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.407664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-tls\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.408095 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.408065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbf0fba-4548-49d5-b602-1522ac7872fb-metrics-client-ca\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.410030 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.409998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.410181 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.410159 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-grpc-tls\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.410254 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.410227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.410345 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.410259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.410410 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.410348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.410507 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.410488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ddbf0fba-4548-49d5-b602-1522ac7872fb-secret-thanos-querier-tls\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.414717 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.414698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz9l\" (UniqueName: \"kubernetes.io/projected/ddbf0fba-4548-49d5-b602-1522ac7872fb-kube-api-access-6jz9l\") pod \"thanos-querier-7d88f99d6d-zk9lv\" (UID: \"ddbf0fba-4548-49d5-b602-1522ac7872fb\") " pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.529162 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.529118 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:50.808620 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:50.808587 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv"] Sep 18 20:05:50.812372 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:50.812341 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbf0fba_4548_49d5_b602_1522ac7872fb.slice/crio-f584d80079f1a67f5fe6a3fd7680c5640a7a447bef6b6acdbf0173a370656fb0 WatchSource:0}: Error finding container f584d80079f1a67f5fe6a3fd7680c5640a7a447bef6b6acdbf0173a370656fb0: Status 404 returned error can't find the container with id f584d80079f1a67f5fe6a3fd7680c5640a7a447bef6b6acdbf0173a370656fb0 Sep 18 20:05:51.114262 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:51.114219 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"f584d80079f1a67f5fe6a3fd7680c5640a7a447bef6b6acdbf0173a370656fb0"} Sep 18 20:05:51.924142 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:51.924107 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns"] Sep 18 20:05:51.928720 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:51.928695 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:51.931397 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:51.931362 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 18 20:05:51.931496 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:51.931474 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-86msf\"" Sep 18 20:05:51.938311 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:51.937579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns"] Sep 18 20:05:52.021736 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.021705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snct2\" (UniqueName: \"kubernetes.io/projected/1b422ad9-fc08-447e-b054-fb9513eaa935-kube-api-access-snct2\") pod \"monitoring-plugin-5984459ff8-8n7ns\" (UID: \"1b422ad9-fc08-447e-b054-fb9513eaa935\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.021857 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.021762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b422ad9-fc08-447e-b054-fb9513eaa935-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-8n7ns\" (UID: \"1b422ad9-fc08-447e-b054-fb9513eaa935\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.122262 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snct2\" (UniqueName: \"kubernetes.io/projected/1b422ad9-fc08-447e-b054-fb9513eaa935-kube-api-access-snct2\") pod \"monitoring-plugin-5984459ff8-8n7ns\" (UID: \"1b422ad9-fc08-447e-b054-fb9513eaa935\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.122725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b422ad9-fc08-447e-b054-fb9513eaa935-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-8n7ns\" (UID: \"1b422ad9-fc08-447e-b054-fb9513eaa935\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.122725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122371 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"444a698ce03ee73ef41458778264a77237188c586ac8f3642e4943e5477a331f"} Sep 18 20:05:52.122725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122452 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"1dd41f3f8564ef2305982a283b1968b3c6cbcd98becdd05be4f42744ebc8fb04"} Sep 18 20:05:52.122725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"d49a1d0b17cc9810f81e0c490515c239708113f102c9216f480cc375bf09cf6e"} Sep 18 20:05:52.122725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"d662213f558c5f2e932aef2dd71c0d88ca0bbe186b84f33b30f7feb98579027f"} Sep 18 20:05:52.122725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.122591 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"fb9c2c5d60ab3c4dd3a5dce59a028a3a076b983cc4161b71cc8dd97ed9459761"} Sep 18 20:05:52.125063 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.125041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b422ad9-fc08-447e-b054-fb9513eaa935-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-8n7ns\" (UID: \"1b422ad9-fc08-447e-b054-fb9513eaa935\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.129594 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.129568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snct2\" (UniqueName: \"kubernetes.io/projected/1b422ad9-fc08-447e-b054-fb9513eaa935-kube-api-access-snct2\") pod \"monitoring-plugin-5984459ff8-8n7ns\" (UID: \"1b422ad9-fc08-447e-b054-fb9513eaa935\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.243183 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.243091 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:52.518380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:52.518299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns"] Sep 18 20:05:52.783536 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:05:52.783460 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b422ad9_fc08_447e_b054_fb9513eaa935.slice/crio-a8591c0e8cc24ccaee6c59b27011a48373ba66f86045dc6809eb9672814e0730 WatchSource:0}: Error finding container a8591c0e8cc24ccaee6c59b27011a48373ba66f86045dc6809eb9672814e0730: Status 404 returned error can't find the container with id a8591c0e8cc24ccaee6c59b27011a48373ba66f86045dc6809eb9672814e0730 Sep 18 20:05:53.129213 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:53.129170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"937afec6068272f6e3c84320f287d90bba52f2ef58359f9d44668d2f0636407c"} Sep 18 20:05:53.129725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:53.129314 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"c279680138c214ea8cc7a4f6e7889a67fa5cae49e6bf21d12d645167075e8909"} Sep 18 20:05:53.129725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:53.129358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"da9946c3c105353c4d53805754a2ba42cc0f99ee1019e95993aafa9350601698"} Sep 18 20:05:53.130664 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:53.130631 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" event={"ID":"1b422ad9-fc08-447e-b054-fb9513eaa935","Type":"ContainerStarted","Data":"a8591c0e8cc24ccaee6c59b27011a48373ba66f86045dc6809eb9672814e0730"} Sep 18 20:05:54.138900 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.138860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c7194915-7361-4d15-8e63-4b9243aaa185","Type":"ContainerStarted","Data":"047a969dff88f726459ed6681b429e1409f9a06ffb2424a6f2ad9e2ac1d5308f"} Sep 18 20:05:54.142545 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.142501 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"0a5f7e3ed270101854dde773d8d0c9e16bd37b0227a58c1ba9476b8b59a7fc44"} Sep 18 20:05:54.142685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.142580 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"5bbb5178c6da5e42e6a3be0bb44064bf9f30257c6ff3f63cafddf42f237b263a"} Sep 18 20:05:54.142685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.142624 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" event={"ID":"ddbf0fba-4548-49d5-b602-1522ac7872fb","Type":"ContainerStarted","Data":"6e9a6a70f815badc50afe3f27b3dfde789df11a9353f52faf12b906098e67bb0"} Sep 18 20:05:54.142804 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.142694 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:05:54.167430 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.167356 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.245538198 podStartE2EDuration="6.167334519s" totalImagesPullingTime="3.904989626s" totalInitContainerRuntime="16.806695ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:54.164748867 +0000 UTC m=+273.215109348" watchObservedRunningTime="2026-09-18 20:05:54.167334519 +0000 UTC m=+273.217694968" Sep 18 20:05:54.187102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:54.187031 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" podStartSLOduration=1.852249396 podStartE2EDuration="4.187004791s" totalImagesPullingTime="2.334755395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:54.184430693 +0000 UTC m=+273.234791142" watchObservedRunningTime="2026-09-18 20:05:54.187004791 +0000 UTC m=+273.237365240" Sep 18 20:05:55.148619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:55.148577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" event={"ID":"1b422ad9-fc08-447e-b054-fb9513eaa935","Type":"ContainerStarted","Data":"80ff3b9a0917685b9005ff6d3f8fde36c0d7b45102ec9bb465a942b4a6a23ef4"} Sep 18 20:05:55.149314 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:55.149289 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:55.154519 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:55.154492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" Sep 18 20:05:55.168941 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:55.168893 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n7ns" podStartSLOduration=2.51864405 podStartE2EDuration="4.168877065s" totalImagesPullingTime="1.650233015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:55.167854973 +0000 UTC m=+274.218215421" watchObservedRunningTime="2026-09-18 20:05:55.168877065 +0000 UTC m=+274.219237564" Sep 18 20:05:59.472204 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.472163 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bd4dddcf-56zx6"] Sep 18 20:05:59.503662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.503631 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74fbc8d667-6v65s"] Sep 18 20:05:59.508119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.508099 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.517143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.517115 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fbc8d667-6v65s"] Sep 18 20:05:59.583097 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-oauth-config\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.583097 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-service-ca\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.583353 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-serving-cert\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.583353 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-trusted-ca-bundle\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.583353 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583250 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-oauth-serving-cert\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.583446 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-console-config\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.583446 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.583379 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qkg\" (UniqueName: \"kubernetes.io/projected/8fdea1ed-1a7c-4160-a079-def257a317b1-kube-api-access-j2qkg\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684077 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-console-config\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684119 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qkg\" (UniqueName: \"kubernetes.io/projected/8fdea1ed-1a7c-4160-a079-def257a317b1-kube-api-access-j2qkg\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-oauth-config\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-service-ca\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-serving-cert\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-trusted-ca-bundle\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-oauth-serving-cert\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.684916 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-console-config\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.685017 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.684936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-service-ca\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.685070 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.685050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-oauth-serving-cert\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.685246 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.685221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-trusted-ca-bundle\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.686844 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.686820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-serving-cert\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.686910 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.686859 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-oauth-config\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.692087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.692059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qkg\" (UniqueName: \"kubernetes.io/projected/8fdea1ed-1a7c-4160-a079-def257a317b1-kube-api-access-j2qkg\") pod \"console-74fbc8d667-6v65s\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:05:59.817515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:05:59.817407 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:06:00.093883 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:00.093804 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fbc8d667-6v65s"] Sep 18 20:06:00.096862 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:06:00.096833 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdea1ed_1a7c_4160_a079_def257a317b1.slice/crio-2eef91434787b1327a9e1c422054eaf32bef196cdb35c7263a2a2e1495e418dd WatchSource:0}: Error finding container 2eef91434787b1327a9e1c422054eaf32bef196cdb35c7263a2a2e1495e418dd: Status 404 returned error can't find the container with id 2eef91434787b1327a9e1c422054eaf32bef196cdb35c7263a2a2e1495e418dd Sep 18 20:06:00.155316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:00.155269 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d88f99d6d-zk9lv" Sep 18 20:06:00.169669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:00.169637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc8d667-6v65s" event={"ID":"8fdea1ed-1a7c-4160-a079-def257a317b1","Type":"ContainerStarted","Data":"7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4"} Sep 18 20:06:00.169831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:00.169696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc8d667-6v65s" event={"ID":"8fdea1ed-1a7c-4160-a079-def257a317b1","Type":"ContainerStarted","Data":"2eef91434787b1327a9e1c422054eaf32bef196cdb35c7263a2a2e1495e418dd"} Sep 18 20:06:00.191974 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:00.191922 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74fbc8d667-6v65s" podStartSLOduration=1.191903363 podStartE2EDuration="1.191903363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:00.191607722 +0000 UTC m=+279.241968172" watchObservedRunningTime="2026-09-18 20:06:00.191903363 +0000 UTC m=+279.242263812" Sep 18 20:06:02.736455 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:06:02.736408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-krmvc" podUID="bb2e0673-cd29-40b6-b85c-15851a3174e6" Sep 18 20:06:03.182457 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:03.182421 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-krmvc" Sep 18 20:06:06.540425 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:06.540380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:06:06.542784 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:06.542761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2e0673-cd29-40b6-b85c-15851a3174e6-metrics-tls\") pod \"dns-default-krmvc\" (UID: \"bb2e0673-cd29-40b6-b85c-15851a3174e6\") " pod="openshift-dns/dns-default-krmvc" Sep 18 20:06:06.787879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:06.787837 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dw2ht\"" Sep 18 20:06:06.794374 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:06.794296 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-krmvc" Sep 18 20:06:07.068980 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:07.068894 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-krmvc"] Sep 18 20:06:07.071991 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:06:07.071958 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2e0673_cd29_40b6_b85c_15851a3174e6.slice/crio-c446f8bd9b08b37a095c6f584b0f35f8c9a6258234517cb5c31622861bbbef84 WatchSource:0}: Error finding container c446f8bd9b08b37a095c6f584b0f35f8c9a6258234517cb5c31622861bbbef84: Status 404 returned error can't find the container with id c446f8bd9b08b37a095c6f584b0f35f8c9a6258234517cb5c31622861bbbef84 Sep 18 20:06:07.199267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:07.199231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-krmvc" event={"ID":"bb2e0673-cd29-40b6-b85c-15851a3174e6","Type":"ContainerStarted","Data":"c446f8bd9b08b37a095c6f584b0f35f8c9a6258234517cb5c31622861bbbef84"} Sep 18 20:06:09.208396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.208297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-krmvc" event={"ID":"bb2e0673-cd29-40b6-b85c-15851a3174e6","Type":"ContainerStarted","Data":"0590dfb0e81c072c4f4ecf570371a07ff9b80c91e5aceea2668349f399d52cc0"} Sep 18 20:06:09.208396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.208352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-krmvc" event={"ID":"bb2e0673-cd29-40b6-b85c-15851a3174e6","Type":"ContainerStarted","Data":"84faa9e57fd9c67b529f9b6c05614e37982eae05088336492e6579898b059b6d"} Sep 18 20:06:09.208396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.208391 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-krmvc" Sep 18 20:06:09.226669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.226614 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-krmvc" podStartSLOduration=251.442487647 podStartE2EDuration="4m13.226596947s" totalImagesPullingTime="1.7841093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:09.225605112 +0000 UTC m=+288.275965560" watchObservedRunningTime="2026-09-18 20:06:09.226596947 +0000 UTC m=+288.276957395" Sep 18 20:06:09.817768 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.817727 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:06:09.818034 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.817800 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:06:09.822752 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:09.822721 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:06:10.215753 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:10.215716 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:06:19.214705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:19.214667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-krmvc" Sep 18 20:06:23.088102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:23.088074 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:06:23.088663 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:23.088190 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:06:23.095820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:23.095784 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 20:06:24.491690 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.491614 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9bd4dddcf-56zx6" podUID="567e59db-5c42-4881-b981-6c2fcfbb7198" containerName="console" containerID="cri-o://083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2" gracePeriod=15 Sep 18 20:06:24.755475 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.755400 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:06:24.895871 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.895829 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-console-config\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.895871 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.895876 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzwgz\" (UniqueName: \"kubernetes.io/projected/567e59db-5c42-4881-b981-6c2fcfbb7198-kube-api-access-lzwgz\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.896115 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.895910 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-oauth-serving-cert\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.896115 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.895942 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-service-ca\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.896115 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.895977 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-serving-cert\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.896115 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.896069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-oauth-config\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.896115 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.896098 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-trusted-ca-bundle\") pod \"567e59db-5c42-4881-b981-6c2fcfbb7198\" (UID: \"567e59db-5c42-4881-b981-6c2fcfbb7198\") " Sep 18 20:06:24.896387 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.896353 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-console-config" (OuterVolumeSpecName: "console-config") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:06:24.896445 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.896413 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:06:24.896654 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.896630 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:06:24.896773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.896652 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-service-ca" (OuterVolumeSpecName: "service-ca") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:06:24.898438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.898411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:06:24.898619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.898591 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e59db-5c42-4881-b981-6c2fcfbb7198-kube-api-access-lzwgz" (OuterVolumeSpecName: "kube-api-access-lzwgz") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "kube-api-access-lzwgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:06:24.898707 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.898631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "567e59db-5c42-4881-b981-6c2fcfbb7198" (UID: "567e59db-5c42-4881-b981-6c2fcfbb7198"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:06:24.997201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997169 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-oauth-config\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:24.997201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997195 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-trusted-ca-bundle\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:24.997201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997205 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-console-config\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:24.997451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997214 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwgz\" (UniqueName: \"kubernetes.io/projected/567e59db-5c42-4881-b981-6c2fcfbb7198-kube-api-access-lzwgz\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:24.997451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997223 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-oauth-serving-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:24.997451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997232 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/567e59db-5c42-4881-b981-6c2fcfbb7198-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:24.997451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:24.997240 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/567e59db-5c42-4881-b981-6c2fcfbb7198-console-serving-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:06:25.273416 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.273385 2559 generic.go:356] "Generic (PLEG): container finished" podID="567e59db-5c42-4881-b981-6c2fcfbb7198" containerID="083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2" exitCode=0 Sep 18 20:06:25.273592 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.273445 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bd4dddcf-56zx6" Sep 18 20:06:25.273592 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.273478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bd4dddcf-56zx6" event={"ID":"567e59db-5c42-4881-b981-6c2fcfbb7198","Type":"ContainerDied","Data":"083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2"} Sep 18 20:06:25.273592 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.273526 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bd4dddcf-56zx6" event={"ID":"567e59db-5c42-4881-b981-6c2fcfbb7198","Type":"ContainerDied","Data":"47afa924ccfe8dfe941dc0fb958b9cabffccbcbbced3ed53f8e701ceec160fdd"} Sep 18 20:06:25.273592 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.273546 2559 scope.go:117] "RemoveContainer" containerID="083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2" Sep 18 20:06:25.281762 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.281745 2559 scope.go:117] "RemoveContainer" containerID="083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2" Sep 18 20:06:25.282057 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:06:25.282033 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2\": container with ID starting with 083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2 not found: ID does not exist" containerID="083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2" Sep 18 20:06:25.282135 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.282070 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2"} err="failed to get container status \"083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2\": rpc error: code = NotFound desc = could not find container \"083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2\": container with ID starting with 083543788affdc4b60b65ee73233c0f8cc845a2d0b08e1754fc378d94feffec2 not found: ID does not exist" Sep 18 20:06:25.289827 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.289800 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bd4dddcf-56zx6"] Sep 18 20:06:25.292702 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:25.292678 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9bd4dddcf-56zx6"] Sep 18 20:06:27.182895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:27.182859 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567e59db-5c42-4881-b981-6c2fcfbb7198" path="/var/lib/kubelet/pods/567e59db-5c42-4881-b981-6c2fcfbb7198/volumes" Sep 18 20:06:33.314938 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:33.314892 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" podUID="f17fd22f-7002-4103-a94d-a72a7f0eb453" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 18 20:06:43.314719 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:43.314678 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" podUID="f17fd22f-7002-4103-a94d-a72a7f0eb453" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 18 20:06:53.314851 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:53.314813 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" podUID="f17fd22f-7002-4103-a94d-a72a7f0eb453" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 18 20:06:53.315246 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:53.314879 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" Sep 18 20:06:53.315383 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:53.315365 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8476670c0760206b5f48a833b79295d33bb9ab4b80aa0068c3551a911ed39ad2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 18 20:06:53.315422 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:53.315402 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" podUID="f17fd22f-7002-4103-a94d-a72a7f0eb453" containerName="service-proxy" containerID="cri-o://8476670c0760206b5f48a833b79295d33bb9ab4b80aa0068c3551a911ed39ad2" gracePeriod=30 Sep 18 20:06:54.402007 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:54.401973 2559 generic.go:356] "Generic (PLEG): container finished" podID="f17fd22f-7002-4103-a94d-a72a7f0eb453" containerID="8476670c0760206b5f48a833b79295d33bb9ab4b80aa0068c3551a911ed39ad2" exitCode=2 Sep 18 20:06:54.402517 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:54.402052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" event={"ID":"f17fd22f-7002-4103-a94d-a72a7f0eb453","Type":"ContainerDied","Data":"8476670c0760206b5f48a833b79295d33bb9ab4b80aa0068c3551a911ed39ad2"} Sep 18 20:06:54.402517 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:06:54.402110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bddb78f4c-tq2m6" event={"ID":"f17fd22f-7002-4103-a94d-a72a7f0eb453","Type":"ContainerStarted","Data":"6eaa28c3fb574d6ebb559238bd24b693addf9e4be57c60754f944e48e31acc42"} Sep 18 20:07:18.113417 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.113335 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75dfcc5c9d-kdwc6"] Sep 18 20:07:18.113855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.113618 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567e59db-5c42-4881-b981-6c2fcfbb7198" containerName="console" Sep 18 20:07:18.113855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.113629 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="567e59db-5c42-4881-b981-6c2fcfbb7198" containerName="console" Sep 18 20:07:18.113855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.113697 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="567e59db-5c42-4881-b981-6c2fcfbb7198" containerName="console" Sep 18 20:07:18.116503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.116487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.128540 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.128515 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75dfcc5c9d-kdwc6"] Sep 18 20:07:18.187821 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.187784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w547p\" (UniqueName: \"kubernetes.io/projected/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-kube-api-access-w547p\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.188023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.187831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-oauth-serving-cert\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.188023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.187853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-config\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.188023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.187934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-trusted-ca-bundle\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.188023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.187977 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-serving-cert\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.188217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.188046 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-oauth-config\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.188217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.188119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-service-ca\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-oauth-serving-cert\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-config\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-trusted-ca-bundle\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-serving-cert\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-oauth-config\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-service-ca\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.289503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.289435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w547p\" (UniqueName: \"kubernetes.io/projected/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-kube-api-access-w547p\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.290064 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.290022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-config\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.290184 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.290157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-oauth-serving-cert\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.290255 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.290237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-service-ca\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.290331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.290308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-trusted-ca-bundle\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.291848 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.291817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-oauth-config\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.291951 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.291902 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-serving-cert\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.297905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.297884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w547p\" (UniqueName: \"kubernetes.io/projected/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-kube-api-access-w547p\") pod \"console-75dfcc5c9d-kdwc6\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.425490 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.425398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:18.699854 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:18.699758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75dfcc5c9d-kdwc6"] Sep 18 20:07:18.702651 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:07:18.702625 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfa5dde_3d55_459d_9eb5_250919bf4bc8.slice/crio-4ea00a5a7a93c668f5b641d4bfd7524403aebded076a8ccb7fb807d306f59994 WatchSource:0}: Error finding container 4ea00a5a7a93c668f5b641d4bfd7524403aebded076a8ccb7fb807d306f59994: Status 404 returned error can't find the container with id 4ea00a5a7a93c668f5b641d4bfd7524403aebded076a8ccb7fb807d306f59994 Sep 18 20:07:19.499785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:19.499753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dfcc5c9d-kdwc6" event={"ID":"fbfa5dde-3d55-459d-9eb5-250919bf4bc8","Type":"ContainerStarted","Data":"ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5"} Sep 18 20:07:19.500150 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:19.499801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dfcc5c9d-kdwc6" event={"ID":"fbfa5dde-3d55-459d-9eb5-250919bf4bc8","Type":"ContainerStarted","Data":"4ea00a5a7a93c668f5b641d4bfd7524403aebded076a8ccb7fb807d306f59994"} Sep 18 20:07:19.517829 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:19.517777 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75dfcc5c9d-kdwc6" podStartSLOduration=1.517761575 podStartE2EDuration="1.517761575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:19.516438227 +0000 UTC m=+358.566798675" watchObservedRunningTime="2026-09-18 20:07:19.517761575 +0000 UTC m=+358.568122023" Sep 18 20:07:28.425657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:28.425607 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:28.426066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:28.425680 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:28.430321 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:28.430297 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:28.536685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:28.536653 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:07:28.579459 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:28.579423 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fbc8d667-6v65s"] Sep 18 20:07:53.601189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.601135 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74fbc8d667-6v65s" podUID="8fdea1ed-1a7c-4160-a079-def257a317b1" containerName="console" containerID="cri-o://7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4" gracePeriod=15 Sep 18 20:07:53.846833 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.846804 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:07:53.976357 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976246 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-oauth-serving-cert\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976357 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976335 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-serving-cert\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976375 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-service-ca\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976414 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2qkg\" (UniqueName: \"kubernetes.io/projected/8fdea1ed-1a7c-4160-a079-def257a317b1-kube-api-access-j2qkg\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976448 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-console-config\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-oauth-config\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976549 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-trusted-ca-bundle\") pod \"8fdea1ed-1a7c-4160-a079-def257a317b1\" (UID: \"8fdea1ed-1a7c-4160-a079-def257a317b1\") " Sep 18 20:07:53.976801 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976716 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:07:53.977007 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976975 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-oauth-serving-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:53.977135 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.976998 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-console-config" (OuterVolumeSpecName: "console-config") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:07:53.977135 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.977041 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:07:53.977135 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.977063 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:07:53.978626 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.978599 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdea1ed-1a7c-4160-a079-def257a317b1-kube-api-access-j2qkg" (OuterVolumeSpecName: "kube-api-access-j2qkg") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "kube-api-access-j2qkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:07:53.978727 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.978688 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:07:53.978727 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:53.978710 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8fdea1ed-1a7c-4160-a079-def257a317b1" (UID: "8fdea1ed-1a7c-4160-a079-def257a317b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:07:54.077571 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.077529 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-serving-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:54.077571 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.077563 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:54.077571 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.077572 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qkg\" (UniqueName: \"kubernetes.io/projected/8fdea1ed-1a7c-4160-a079-def257a317b1-kube-api-access-j2qkg\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:54.077571 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.077582 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-console-config\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:54.077852 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.077596 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8fdea1ed-1a7c-4160-a079-def257a317b1-console-oauth-config\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:54.077852 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.077605 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fdea1ed-1a7c-4160-a079-def257a317b1-trusted-ca-bundle\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:07:54.633591 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.633551 2559 generic.go:356] "Generic (PLEG): container finished" podID="8fdea1ed-1a7c-4160-a079-def257a317b1" containerID="7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4" exitCode=0 Sep 18 20:07:54.634016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.633615 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc8d667-6v65s" Sep 18 20:07:54.634016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.633616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc8d667-6v65s" event={"ID":"8fdea1ed-1a7c-4160-a079-def257a317b1","Type":"ContainerDied","Data":"7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4"} Sep 18 20:07:54.634016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.633726 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc8d667-6v65s" event={"ID":"8fdea1ed-1a7c-4160-a079-def257a317b1","Type":"ContainerDied","Data":"2eef91434787b1327a9e1c422054eaf32bef196cdb35c7263a2a2e1495e418dd"} Sep 18 20:07:54.634016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.633745 2559 scope.go:117] "RemoveContainer" containerID="7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4" Sep 18 20:07:54.642342 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.642323 2559 scope.go:117] "RemoveContainer" containerID="7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4" Sep 18 20:07:54.642610 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:07:54.642595 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4\": container with ID starting with 7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4 not found: ID does not exist" containerID="7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4" Sep 18 20:07:54.642658 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.642618 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4"} err="failed to get container status \"7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4\": rpc error: code = NotFound desc = could not find container \"7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4\": container with ID starting with 7e0eec20ff4ef917cb71a319f6056992bc5e21dbf77b24c47a0b694d455dd0a4 not found: ID does not exist" Sep 18 20:07:54.654066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.654043 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fbc8d667-6v65s"] Sep 18 20:07:54.657263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:54.657242 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74fbc8d667-6v65s"] Sep 18 20:07:55.182015 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:07:55.181979 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fdea1ed-1a7c-4160-a079-def257a317b1" path="/var/lib/kubelet/pods/8fdea1ed-1a7c-4160-a079-def257a317b1/volumes" Sep 18 20:08:34.256362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.256327 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24"] Sep 18 20:08:34.256831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.256619 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdea1ed-1a7c-4160-a079-def257a317b1" containerName="console" Sep 18 20:08:34.256831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.256630 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdea1ed-1a7c-4160-a079-def257a317b1" containerName="console" Sep 18 20:08:34.256831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.256683 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fdea1ed-1a7c-4160-a079-def257a317b1" containerName="console" Sep 18 20:08:34.260900 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.260882 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.264020 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.263995 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 18 20:08:34.264423 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.264408 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dgw8d\"" Sep 18 20:08:34.265477 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.265459 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 18 20:08:34.269864 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.269842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24"] Sep 18 20:08:34.385072 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.385038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44549c80-4b4f-41e0-a1aa-53f1a130aec6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s2v24\" (UID: \"44549c80-4b4f-41e0-a1aa-53f1a130aec6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.385335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.385124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzb9d\" (UniqueName: \"kubernetes.io/projected/44549c80-4b4f-41e0-a1aa-53f1a130aec6-kube-api-access-dzb9d\") pod \"openshift-lws-operator-fd8ccff4c-s2v24\" (UID: \"44549c80-4b4f-41e0-a1aa-53f1a130aec6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.486494 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.486451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44549c80-4b4f-41e0-a1aa-53f1a130aec6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s2v24\" (UID: \"44549c80-4b4f-41e0-a1aa-53f1a130aec6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.486655 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.486529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzb9d\" (UniqueName: \"kubernetes.io/projected/44549c80-4b4f-41e0-a1aa-53f1a130aec6-kube-api-access-dzb9d\") pod \"openshift-lws-operator-fd8ccff4c-s2v24\" (UID: \"44549c80-4b4f-41e0-a1aa-53f1a130aec6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.486897 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.486873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44549c80-4b4f-41e0-a1aa-53f1a130aec6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s2v24\" (UID: \"44549c80-4b4f-41e0-a1aa-53f1a130aec6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.497316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.497292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzb9d\" (UniqueName: \"kubernetes.io/projected/44549c80-4b4f-41e0-a1aa-53f1a130aec6-kube-api-access-dzb9d\") pod \"openshift-lws-operator-fd8ccff4c-s2v24\" (UID: \"44549c80-4b4f-41e0-a1aa-53f1a130aec6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.569567 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.569480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" Sep 18 20:08:34.842831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.842731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24"] Sep 18 20:08:34.845817 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:08:34.845783 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44549c80_4b4f_41e0_a1aa_53f1a130aec6.slice/crio-2791d16ae9aa80e4701b24208825d6549401db4078d25cb41e4db3ebd6617b71 WatchSource:0}: Error finding container 2791d16ae9aa80e4701b24208825d6549401db4078d25cb41e4db3ebd6617b71: Status 404 returned error can't find the container with id 2791d16ae9aa80e4701b24208825d6549401db4078d25cb41e4db3ebd6617b71 Sep 18 20:08:34.847248 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:34.847226 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:08:35.790352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:35.790309 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" event={"ID":"44549c80-4b4f-41e0-a1aa-53f1a130aec6","Type":"ContainerStarted","Data":"2791d16ae9aa80e4701b24208825d6549401db4078d25cb41e4db3ebd6617b71"} Sep 18 20:08:37.800576 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:37.800484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" event={"ID":"44549c80-4b4f-41e0-a1aa-53f1a130aec6","Type":"ContainerStarted","Data":"3cf388e6843c60645dac36b79a173df2a2b45d3b16e5197d78b45f7377a208e8"} Sep 18 20:08:37.818532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:37.818473 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s2v24" podStartSLOduration=1.244019837 podStartE2EDuration="3.81845171s" totalImagesPullingTime="2.574431873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:08:37.816106693 +0000 UTC m=+436.866467142" watchObservedRunningTime="2026-09-18 20:08:37.81845171 +0000 UTC m=+436.868812157" Sep 18 20:08:55.866233 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.866200 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf"] Sep 18 20:08:55.869695 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.869674 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:55.872740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.872715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ws545\"" Sep 18 20:08:55.872874 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.872743 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 20:08:55.872944 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.872868 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 20:08:55.872944 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.872894 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 20:08:55.873070 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.873055 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 20:08:55.896125 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.896093 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf"] Sep 18 20:08:55.957002 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.956964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-manifests\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:55.957002 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.957007 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-webhook-cert\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:55.957217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.957107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:55.957217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.957161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-tmp\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:55.957311 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:55.957219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbl47\" (UniqueName: \"kubernetes.io/projected/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-kube-api-access-pbl47\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.058623 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.058584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbl47\" (UniqueName: \"kubernetes.io/projected/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-kube-api-access-pbl47\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.058623 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.058630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-manifests\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.058840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.058648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-webhook-cert\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.058840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.058674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.058840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.058709 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-tmp\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.059080 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.059061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-manifests\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.061047 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.061016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-tmp\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.061251 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.061231 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-webhook-cert\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.061354 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.061265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.067758 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.067736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbl47\" (UniqueName: \"kubernetes.io/projected/f584d8bd-54fa-4dde-8ae4-2633a9f36ee8-kube-api-access-pbl47\") pod \"opendatahub-operator-controller-manager-668c7cd96d-swxwf\" (UID: \"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8\") " pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.180116 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.180023 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:08:56.455944 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.455866 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf"] Sep 18 20:08:56.459572 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:08:56.459546 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf584d8bd_54fa_4dde_8ae4_2633a9f36ee8.slice/crio-9af15c2d34aaecdcafc62e6fd531a03a5d85212e78f7220446c321e823d67dbe WatchSource:0}: Error finding container 9af15c2d34aaecdcafc62e6fd531a03a5d85212e78f7220446c321e823d67dbe: Status 404 returned error can't find the container with id 9af15c2d34aaecdcafc62e6fd531a03a5d85212e78f7220446c321e823d67dbe Sep 18 20:08:56.875210 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:56.875175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" event={"ID":"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8","Type":"ContainerStarted","Data":"9af15c2d34aaecdcafc62e6fd531a03a5d85212e78f7220446c321e823d67dbe"} Sep 18 20:08:59.889388 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:08:59.889351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" event={"ID":"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8","Type":"ContainerStarted","Data":"524a5c4817391d67ef007a69d83562d25d6492ea0b031bb3cd2ad8470db170b5"} Sep 18 20:09:00.899095 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:00.897379 2559 generic.go:356] "Generic (PLEG): container finished" podID="f584d8bd-54fa-4dde-8ae4-2633a9f36ee8" containerID="524a5c4817391d67ef007a69d83562d25d6492ea0b031bb3cd2ad8470db170b5" exitCode=0 Sep 18 20:09:00.899095 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:00.897472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" event={"ID":"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8","Type":"ContainerDied","Data":"524a5c4817391d67ef007a69d83562d25d6492ea0b031bb3cd2ad8470db170b5"} Sep 18 20:09:01.904619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:01.904584 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" event={"ID":"f584d8bd-54fa-4dde-8ae4-2633a9f36ee8","Type":"ContainerStarted","Data":"623a04ba2d71d42dd99c754d3c01de813ce11a124127c5f5c4816c3fa96498ab"} Sep 18 20:09:01.905121 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:01.904692 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:09:01.934371 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:01.934317 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" podStartSLOduration=2.535222938 podStartE2EDuration="6.934298173s" totalImagesPullingTime="3.321321942s" totalInitContainerRuntime="1.077753293s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:08:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:01.932375193 +0000 UTC m=+460.982735645" watchObservedRunningTime="2026-09-18 20:09:01.934298173 +0000 UTC m=+460.984658621" Sep 18 20:09:12.911089 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:12.911052 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-668c7cd96d-swxwf" Sep 18 20:09:15.813301 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.813253 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66b8956864-gk67t"] Sep 18 20:09:15.817494 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.817473 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:15.822418 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.822395 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 20:09:15.822536 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.822471 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 18 20:09:15.823286 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.823250 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 18 20:09:15.824170 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.824151 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 20:09:15.824261 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.824197 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7dlvt\"" Sep 18 20:09:15.844927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.844898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66b8956864-gk67t"] Sep 18 20:09:15.918299 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.918246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj88\" (UniqueName: \"kubernetes.io/projected/80355379-fa48-4644-aea3-daef56ee1aa8-kube-api-access-dkj88\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:15.918299 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.918302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80355379-fa48-4644-aea3-daef56ee1aa8-tls-certs\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:15.918493 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:15.918366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80355379-fa48-4644-aea3-daef56ee1aa8-tmp\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.019105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.019066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80355379-fa48-4644-aea3-daef56ee1aa8-tmp\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.019310 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.019155 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj88\" (UniqueName: \"kubernetes.io/projected/80355379-fa48-4644-aea3-daef56ee1aa8-kube-api-access-dkj88\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.019310 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.019181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80355379-fa48-4644-aea3-daef56ee1aa8-tls-certs\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.019389 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:09:16.019309 2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 18 20:09:16.019389 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:09:16.019385 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80355379-fa48-4644-aea3-daef56ee1aa8-tls-certs podName:80355379-fa48-4644-aea3-daef56ee1aa8 nodeName:}" failed. No retries permitted until 2026-09-18 20:09:16.519366941 +0000 UTC m=+475.569727369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/80355379-fa48-4644-aea3-daef56ee1aa8-tls-certs") pod "kube-auth-proxy-66b8956864-gk67t" (UID: "80355379-fa48-4644-aea3-daef56ee1aa8") : secret "kube-auth-proxy-tls" not found Sep 18 20:09:16.021386 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.021355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80355379-fa48-4644-aea3-daef56ee1aa8-tmp\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.033615 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.033586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj88\" (UniqueName: \"kubernetes.io/projected/80355379-fa48-4644-aea3-daef56ee1aa8-kube-api-access-dkj88\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.524317 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.524265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80355379-fa48-4644-aea3-daef56ee1aa8-tls-certs\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.526618 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.526587 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80355379-fa48-4644-aea3-daef56ee1aa8-tls-certs\") pod \"kube-auth-proxy-66b8956864-gk67t\" (UID: \"80355379-fa48-4644-aea3-daef56ee1aa8\") " pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:16.727327 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:16.727292 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" Sep 18 20:09:17.001192 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:17.001156 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66b8956864-gk67t"] Sep 18 20:09:17.004801 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:09:17.004769 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80355379_fa48_4644_aea3_daef56ee1aa8.slice/crio-614d5a3d74c21312e390d7dbc80cfb35f45eddceda3ea6c35f2d317d62e8a06a WatchSource:0}: Error finding container 614d5a3d74c21312e390d7dbc80cfb35f45eddceda3ea6c35f2d317d62e8a06a: Status 404 returned error can't find the container with id 614d5a3d74c21312e390d7dbc80cfb35f45eddceda3ea6c35f2d317d62e8a06a Sep 18 20:09:17.977559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:17.977482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" event={"ID":"80355379-fa48-4644-aea3-daef56ee1aa8","Type":"ContainerStarted","Data":"614d5a3d74c21312e390d7dbc80cfb35f45eddceda3ea6c35f2d317d62e8a06a"} Sep 18 20:09:19.987577 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:19.987494 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" event={"ID":"80355379-fa48-4644-aea3-daef56ee1aa8","Type":"ContainerStarted","Data":"0811886576d6691e377b898f889d9b16c3ba010d1d4f9019aaf15133ad6e6e13"} Sep 18 20:09:20.004231 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:20.004178 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66b8956864-gk67t" podStartSLOduration=2.331219739 podStartE2EDuration="5.004161579s" totalImagesPullingTime="2.67294184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:20.002378102 +0000 UTC m=+479.052738549" watchObservedRunningTime="2026-09-18 20:09:20.004161579 +0000 UTC m=+479.054522026" Sep 18 20:09:26.483495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.483457 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h8d5d"] Sep 18 20:09:26.488380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.488352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:26.490886 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.490860 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 18 20:09:26.491071 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.490932 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-csb5f\"" Sep 18 20:09:26.496471 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.496269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h8d5d"] Sep 18 20:09:26.616101 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.616065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9qx\" (UniqueName: \"kubernetes.io/projected/23612048-c42d-4dc2-96c2-5c10e94b7305-kube-api-access-lw9qx\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:26.616266 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.616127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23612048-c42d-4dc2-96c2-5c10e94b7305-cert\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:26.717074 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.717026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23612048-c42d-4dc2-96c2-5c10e94b7305-cert\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:26.717311 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.717131 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9qx\" (UniqueName: \"kubernetes.io/projected/23612048-c42d-4dc2-96c2-5c10e94b7305-kube-api-access-lw9qx\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:26.717311 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:09:26.717186 2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 18 20:09:26.717311 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:09:26.717266 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23612048-c42d-4dc2-96c2-5c10e94b7305-cert podName:23612048-c42d-4dc2-96c2-5c10e94b7305 nodeName:}" failed. No retries permitted until 2026-09-18 20:09:27.217250244 +0000 UTC m=+486.267610670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/23612048-c42d-4dc2-96c2-5c10e94b7305-cert") pod "odh-model-controller-7998c67dc4-h8d5d" (UID: "23612048-c42d-4dc2-96c2-5c10e94b7305") : secret "odh-model-controller-webhook-cert" not found Sep 18 20:09:26.725511 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:26.725481 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9qx\" (UniqueName: \"kubernetes.io/projected/23612048-c42d-4dc2-96c2-5c10e94b7305-kube-api-access-lw9qx\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:27.002675 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.002638 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wwh6j"] Sep 18 20:09:27.006038 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.006020 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.008758 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.008738 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 18 20:09:27.008873 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.008829 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-l757x\"" Sep 18 20:09:27.016323 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.016296 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wwh6j"] Sep 18 20:09:27.120586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.120547 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60986192-d301-4e5a-ad99-3c63d1667c85-tls-certs\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.120791 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.120627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zdl\" (UniqueName: \"kubernetes.io/projected/60986192-d301-4e5a-ad99-3c63d1667c85-kube-api-access-k4zdl\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.221085 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.221044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zdl\" (UniqueName: \"kubernetes.io/projected/60986192-d301-4e5a-ad99-3c63d1667c85-kube-api-access-k4zdl\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.221301 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.221103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60986192-d301-4e5a-ad99-3c63d1667c85-tls-certs\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.221301 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.221162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23612048-c42d-4dc2-96c2-5c10e94b7305-cert\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:27.221301 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:09:27.221263 2559 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 18 20:09:27.221423 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:09:27.221360 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60986192-d301-4e5a-ad99-3c63d1667c85-tls-certs podName:60986192-d301-4e5a-ad99-3c63d1667c85 nodeName:}" failed. No retries permitted until 2026-09-18 20:09:27.721339811 +0000 UTC m=+486.771700261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/60986192-d301-4e5a-ad99-3c63d1667c85-tls-certs") pod "model-serving-api-54d89bd5fc-wwh6j" (UID: "60986192-d301-4e5a-ad99-3c63d1667c85") : secret "model-serving-api-tls" not found Sep 18 20:09:27.223709 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.223685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23612048-c42d-4dc2-96c2-5c10e94b7305-cert\") pod \"odh-model-controller-7998c67dc4-h8d5d\" (UID: \"23612048-c42d-4dc2-96c2-5c10e94b7305\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:27.232393 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.232362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zdl\" (UniqueName: \"kubernetes.io/projected/60986192-d301-4e5a-ad99-3c63d1667c85-kube-api-access-k4zdl\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.400923 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.400889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:27.678935 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.678903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h8d5d"] Sep 18 20:09:27.680614 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:09:27.680581 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23612048_c42d_4dc2_96c2_5c10e94b7305.slice/crio-b62ce3bbd1fe3b78505c369c9e2be4bbe329864b64515e3a6d20c3b6d75fd5fa WatchSource:0}: Error finding container b62ce3bbd1fe3b78505c369c9e2be4bbe329864b64515e3a6d20c3b6d75fd5fa: Status 404 returned error can't find the container with id b62ce3bbd1fe3b78505c369c9e2be4bbe329864b64515e3a6d20c3b6d75fd5fa Sep 18 20:09:27.724659 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.724624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60986192-d301-4e5a-ad99-3c63d1667c85-tls-certs\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.727308 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.727246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60986192-d301-4e5a-ad99-3c63d1667c85-tls-certs\") pod \"model-serving-api-54d89bd5fc-wwh6j\" (UID: \"60986192-d301-4e5a-ad99-3c63d1667c85\") " pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:27.917807 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:27.917749 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:28.028295 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:28.028241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" event={"ID":"23612048-c42d-4dc2-96c2-5c10e94b7305","Type":"ContainerStarted","Data":"b62ce3bbd1fe3b78505c369c9e2be4bbe329864b64515e3a6d20c3b6d75fd5fa"} Sep 18 20:09:28.192799 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:28.192722 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wwh6j"] Sep 18 20:09:28.195541 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:09:28.195514 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60986192_d301_4e5a_ad99_3c63d1667c85.slice/crio-739550a72ad969721f24a618af690ab067dfa653fd04a8546b3700beb7949697 WatchSource:0}: Error finding container 739550a72ad969721f24a618af690ab067dfa653fd04a8546b3700beb7949697: Status 404 returned error can't find the container with id 739550a72ad969721f24a618af690ab067dfa653fd04a8546b3700beb7949697 Sep 18 20:09:29.036400 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:29.036356 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" event={"ID":"60986192-d301-4e5a-ad99-3c63d1667c85","Type":"ContainerStarted","Data":"739550a72ad969721f24a618af690ab067dfa653fd04a8546b3700beb7949697"} Sep 18 20:09:32.054088 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:32.054034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" event={"ID":"23612048-c42d-4dc2-96c2-5c10e94b7305","Type":"ContainerStarted","Data":"954eb6a8ec0c3a6e4d4a02eb8662fca9658b5dd5cbb374239a6d7efb41109a80"} Sep 18 20:09:32.054576 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:32.054216 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:32.055783 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:32.055750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" event={"ID":"60986192-d301-4e5a-ad99-3c63d1667c85","Type":"ContainerStarted","Data":"e691c934f0391ffb78f1ad10bbf5fee1f17d3d4aab5da5c2c5f959e7484c2c10"} Sep 18 20:09:32.055918 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:32.055906 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:09:32.071292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:32.071228 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" podStartSLOduration=2.683607647 podStartE2EDuration="6.071213126s" totalImagesPullingTime="3.387605479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:32.07019409 +0000 UTC m=+491.120554537" watchObservedRunningTime="2026-09-18 20:09:32.071213126 +0000 UTC m=+491.121573574" Sep 18 20:09:32.092469 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:32.092406 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" podStartSLOduration=3.2773633540000002 podStartE2EDuration="6.092387461s" totalImagesPullingTime="2.815024107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:32.091190256 +0000 UTC m=+491.141550721" watchObservedRunningTime="2026-09-18 20:09:32.092387461 +0000 UTC m=+491.142747910" Sep 18 20:09:33.874248 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.874209 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg"] Sep 18 20:09:33.885843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.885802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.888586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.888561 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 18 20:09:33.888734 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.888566 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 18 20:09:33.888734 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.888646 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 18 20:09:33.888734 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.888684 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 18 20:09:33.888905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.888871 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg"] Sep 18 20:09:33.888988 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.888970 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zgmmj\"" Sep 18 20:09:33.981938 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.981909 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.982124 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.981949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.982124 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.981982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.982124 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.982043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.982245 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.982118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.982245 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.982146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:33.982245 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:33.982172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97j7v\" (UniqueName: \"kubernetes.io/projected/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-kube-api-access-97j7v\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083557 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083913 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083913 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.083913 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.083828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97j7v\" (UniqueName: \"kubernetes.io/projected/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-kube-api-access-97j7v\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.084643 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.084461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.086064 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.086043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.086228 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.086204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.086434 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.086417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.086489 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.086436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.092247 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.092219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.092247 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.092247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97j7v\" (UniqueName: \"kubernetes.io/projected/3eea43bd-e794-42d0-a8b9-aaddb194dcf8-kube-api-access-97j7v\") pod \"istiod-openshift-gateway-5677ccb49b-bw7lg\" (UID: \"3eea43bd-e794-42d0-a8b9-aaddb194dcf8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.196499 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.196419 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:34.487049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:34.486909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg"] Sep 18 20:09:34.489599 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:09:34.489562 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eea43bd_e794_42d0_a8b9_aaddb194dcf8.slice/crio-fe80789c03a4f2904164f36eb279f6bc4f7563f10d54e07882259ebf27ec28a9 WatchSource:0}: Error finding container fe80789c03a4f2904164f36eb279f6bc4f7563f10d54e07882259ebf27ec28a9: Status 404 returned error can't find the container with id fe80789c03a4f2904164f36eb279f6bc4f7563f10d54e07882259ebf27ec28a9 Sep 18 20:09:35.072520 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:35.072485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" event={"ID":"3eea43bd-e794-42d0-a8b9-aaddb194dcf8","Type":"ContainerStarted","Data":"fe80789c03a4f2904164f36eb279f6bc4f7563f10d54e07882259ebf27ec28a9"} Sep 18 20:09:36.971382 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:36.971339 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 20:09:36.971699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:36.971413 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 20:09:37.085110 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:37.085072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" event={"ID":"3eea43bd-e794-42d0-a8b9-aaddb194dcf8","Type":"ContainerStarted","Data":"f24932448b5a303972d166fc21ddf575cf55dff9eaf943e44bf6df8cfee2ae47"} Sep 18 20:09:37.085307 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:37.085259 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:37.111588 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:37.111527 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" podStartSLOduration=1.632300538 podStartE2EDuration="4.111510242s" totalImagesPullingTime="2.479209704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:37.110744898 +0000 UTC m=+496.161105347" watchObservedRunningTime="2026-09-18 20:09:37.111510242 +0000 UTC m=+496.161870690" Sep 18 20:09:38.091889 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:38.091852 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bw7lg" Sep 18 20:09:43.064496 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:43.064464 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h8d5d" Sep 18 20:09:43.066348 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:09:43.066324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-wwh6j" Sep 18 20:10:44.211187 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.211152 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qwthl"] Sep 18 20:10:44.213175 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.213159 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:10:44.215981 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.215951 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 18 20:10:44.215981 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.215965 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 18 20:10:44.216130 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.216049 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z9qnf\"" Sep 18 20:10:44.224936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.224911 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qwthl"] Sep 18 20:10:44.285035 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.284992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxj6\" (UniqueName: \"kubernetes.io/projected/b1b549ca-85b2-4e54-8c2b-6a7ac4052f02-kube-api-access-8lxj6\") pod \"authorino-operator-76b478bd78-qwthl\" (UID: \"b1b549ca-85b2-4e54-8c2b-6a7ac4052f02\") " pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:10:44.386352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.386319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxj6\" (UniqueName: \"kubernetes.io/projected/b1b549ca-85b2-4e54-8c2b-6a7ac4052f02-kube-api-access-8lxj6\") pod \"authorino-operator-76b478bd78-qwthl\" (UID: \"b1b549ca-85b2-4e54-8c2b-6a7ac4052f02\") " pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:10:44.405408 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.405379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxj6\" (UniqueName: \"kubernetes.io/projected/b1b549ca-85b2-4e54-8c2b-6a7ac4052f02-kube-api-access-8lxj6\") pod \"authorino-operator-76b478bd78-qwthl\" (UID: \"b1b549ca-85b2-4e54-8c2b-6a7ac4052f02\") " pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:10:44.524139 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.524038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:10:44.795637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:44.795526 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qwthl"] Sep 18 20:10:44.798083 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:10:44.798059 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b549ca_85b2_4e54_8c2b_6a7ac4052f02.slice/crio-2358253f21f0eab4d0094dee4e3547c78d7a92a20412d99fb04ca0db960f5a9a WatchSource:0}: Error finding container 2358253f21f0eab4d0094dee4e3547c78d7a92a20412d99fb04ca0db960f5a9a: Status 404 returned error can't find the container with id 2358253f21f0eab4d0094dee4e3547c78d7a92a20412d99fb04ca0db960f5a9a Sep 18 20:10:45.386237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:45.386197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" event={"ID":"b1b549ca-85b2-4e54-8c2b-6a7ac4052f02","Type":"ContainerStarted","Data":"2358253f21f0eab4d0094dee4e3547c78d7a92a20412d99fb04ca0db960f5a9a"} Sep 18 20:10:50.551198 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.551160 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b"] Sep 18 20:10:50.564663 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.564629 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:10:50.567506 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.567480 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fdk2m\"" Sep 18 20:10:50.576877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.576827 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b"] Sep 18 20:10:50.639172 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.639140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nsl\" (UniqueName: \"kubernetes.io/projected/11f32005-2039-4ec5-9762-ee18de5a84e0-kube-api-access-86nsl\") pod \"limitador-operator-controller-manager-74495c69f8-g7q4b\" (UID: \"11f32005-2039-4ec5-9762-ee18de5a84e0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:10:50.739632 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.739595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nsl\" (UniqueName: \"kubernetes.io/projected/11f32005-2039-4ec5-9762-ee18de5a84e0-kube-api-access-86nsl\") pod \"limitador-operator-controller-manager-74495c69f8-g7q4b\" (UID: \"11f32005-2039-4ec5-9762-ee18de5a84e0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:10:50.754410 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.754375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nsl\" (UniqueName: \"kubernetes.io/projected/11f32005-2039-4ec5-9762-ee18de5a84e0-kube-api-access-86nsl\") pod \"limitador-operator-controller-manager-74495c69f8-g7q4b\" (UID: \"11f32005-2039-4ec5-9762-ee18de5a84e0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:10:50.879672 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:50.879627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:10:51.165576 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:51.165542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b"] Sep 18 20:10:51.168113 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:10:51.168085 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f32005_2039_4ec5_9762_ee18de5a84e0.slice/crio-4541edbf710e11e1126261794a34a11f94fef06f19432b0db08aada33ec50ad4 WatchSource:0}: Error finding container 4541edbf710e11e1126261794a34a11f94fef06f19432b0db08aada33ec50ad4: Status 404 returned error can't find the container with id 4541edbf710e11e1126261794a34a11f94fef06f19432b0db08aada33ec50ad4 Sep 18 20:10:51.418543 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:51.418451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" event={"ID":"11f32005-2039-4ec5-9762-ee18de5a84e0","Type":"ContainerStarted","Data":"4541edbf710e11e1126261794a34a11f94fef06f19432b0db08aada33ec50ad4"} Sep 18 20:10:51.419309 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:51.419261 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75dfcc5c9d-kdwc6"] Sep 18 20:10:51.420363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:51.420325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" event={"ID":"b1b549ca-85b2-4e54-8c2b-6a7ac4052f02","Type":"ContainerStarted","Data":"c8a233e41cd97065f59feb617cdc67dc03ba33a00a9828a3de235b218075ae2a"} Sep 18 20:10:51.420495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:51.420475 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:10:51.463437 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:51.463374 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" podStartSLOduration=1.301238502 podStartE2EDuration="7.463355187s" totalImagesPullingTime="6.162116685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:51.46161995 +0000 UTC m=+570.511980400" watchObservedRunningTime="2026-09-18 20:10:51.463355187 +0000 UTC m=+570.513715635" Sep 18 20:10:53.433004 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:53.432974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" event={"ID":"11f32005-2039-4ec5-9762-ee18de5a84e0","Type":"ContainerStarted","Data":"522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24"} Sep 18 20:10:53.433410 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:53.433050 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:10:53.452267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:10:53.452218 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" podStartSLOduration=1.579204016 podStartE2EDuration="3.452200292s" totalImagesPullingTime="1.872996276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:53.451860027 +0000 UTC m=+572.502220478" watchObservedRunningTime="2026-09-18 20:10:53.452200292 +0000 UTC m=+572.502560734" Sep 18 20:11:02.429049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:02.429015 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-qwthl" Sep 18 20:11:04.439604 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:04.439559 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:11:06.544625 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.544590 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b"] Sep 18 20:11:06.545189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.544824 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" containerName="manager" containerID="cri-o://522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24" gracePeriod=2 Sep 18 20:11:06.547260 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:11:06.547218 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g7q4b\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:11:06.547399 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.547264 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b"] Sep 18 20:11:06.568864 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.568830 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2"] Sep 18 20:11:06.569464 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.569440 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" containerName="manager" Sep 18 20:11:06.569541 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.569468 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" containerName="manager" Sep 18 20:11:06.569605 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.569562 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" containerName="manager" Sep 18 20:11:06.572869 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.572847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:06.575247 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:11:06.575215 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g7q4b\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:11:06.588053 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.588022 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2"] Sep 18 20:11:06.672116 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.672081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/7452724f-7913-4a39-aa7b-d20b755b8c1d-kube-api-access-vqwrq\") pod \"limitador-operator-controller-manager-74495c69f8-wbbj2\" (UID: \"7452724f-7913-4a39-aa7b-d20b755b8c1d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:06.773606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.773575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/7452724f-7913-4a39-aa7b-d20b755b8c1d-kube-api-access-vqwrq\") pod \"limitador-operator-controller-manager-74495c69f8-wbbj2\" (UID: \"7452724f-7913-4a39-aa7b-d20b755b8c1d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:06.785644 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.785622 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:11:06.791453 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.786255 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/7452724f-7913-4a39-aa7b-d20b755b8c1d-kube-api-access-vqwrq\") pod \"limitador-operator-controller-manager-74495c69f8-wbbj2\" (UID: \"7452724f-7913-4a39-aa7b-d20b755b8c1d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:06.791889 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:11:06.791858 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g7q4b\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:11:06.874596 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.874552 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86nsl\" (UniqueName: \"kubernetes.io/projected/11f32005-2039-4ec5-9762-ee18de5a84e0-kube-api-access-86nsl\") pod \"11f32005-2039-4ec5-9762-ee18de5a84e0\" (UID: \"11f32005-2039-4ec5-9762-ee18de5a84e0\") " Sep 18 20:11:06.876795 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.876766 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f32005-2039-4ec5-9762-ee18de5a84e0-kube-api-access-86nsl" (OuterVolumeSpecName: "kube-api-access-86nsl") pod "11f32005-2039-4ec5-9762-ee18de5a84e0" (UID: "11f32005-2039-4ec5-9762-ee18de5a84e0"). InnerVolumeSpecName "kube-api-access-86nsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:11:06.925952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.925915 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:06.976191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:06.975496 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nsl\" (UniqueName: \"kubernetes.io/projected/11f32005-2039-4ec5-9762-ee18de5a84e0-kube-api-access-86nsl\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:07.183622 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.183552 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f32005-2039-4ec5-9762-ee18de5a84e0" path="/var/lib/kubelet/pods/11f32005-2039-4ec5-9762-ee18de5a84e0/volumes" Sep 18 20:11:07.202872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.202844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2"] Sep 18 20:11:07.204910 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:11:07.204882 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7452724f_7913_4a39_aa7b_d20b755b8c1d.slice/crio-be301ceb18e95b717d86038728ad1d79455957669c448c11863a82defc7a9c00 WatchSource:0}: Error finding container be301ceb18e95b717d86038728ad1d79455957669c448c11863a82defc7a9c00: Status 404 returned error can't find the container with id be301ceb18e95b717d86038728ad1d79455957669c448c11863a82defc7a9c00 Sep 18 20:11:07.499450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.499358 2559 generic.go:356] "Generic (PLEG): container finished" podID="11f32005-2039-4ec5-9762-ee18de5a84e0" containerID="522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24" exitCode=0 Sep 18 20:11:07.499450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.499431 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g7q4b" Sep 18 20:11:07.499674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.499473 2559 scope.go:117] "RemoveContainer" containerID="522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24" Sep 18 20:11:07.501335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.501303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" event={"ID":"7452724f-7913-4a39-aa7b-d20b755b8c1d","Type":"ContainerStarted","Data":"1d6673810394a7fcfcfe3586d30aa580d1baaf33202ee971b0323397df3e0c8f"} Sep 18 20:11:07.501475 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.501360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" event={"ID":"7452724f-7913-4a39-aa7b-d20b755b8c1d","Type":"ContainerStarted","Data":"be301ceb18e95b717d86038728ad1d79455957669c448c11863a82defc7a9c00"} Sep 18 20:11:07.501475 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.501432 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:07.507638 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.507619 2559 scope.go:117] "RemoveContainer" containerID="522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24" Sep 18 20:11:07.507889 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:11:07.507871 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24\": container with ID starting with 522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24 not found: ID does not exist" containerID="522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24" Sep 18 20:11:07.507947 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.507896 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24"} err="failed to get container status \"522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24\": rpc error: code = NotFound desc = could not find container \"522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24\": container with ID starting with 522fc30f39bfcf5f0e2ab69a881072f61fb2490826933fb3e51114ad99afef24 not found: ID does not exist" Sep 18 20:11:07.526667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:07.526613 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" podStartSLOduration=1.526595076 podStartE2EDuration="1.526595076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:07.525378144 +0000 UTC m=+586.575738591" watchObservedRunningTime="2026-09-18 20:11:07.526595076 +0000 UTC m=+586.576955529" Sep 18 20:11:16.438492 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.438416 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75dfcc5c9d-kdwc6" podUID="fbfa5dde-3d55-459d-9eb5-250919bf4bc8" containerName="console" containerID="cri-o://ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5" gracePeriod=15 Sep 18 20:11:16.686520 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.686494 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:11:16.763151 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763057 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w547p\" (UniqueName: \"kubernetes.io/projected/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-kube-api-access-w547p\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763151 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-serving-cert\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763151 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763134 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-trusted-ca-bundle\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763166 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-config\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763194 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-oauth-serving-cert\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763224 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-service-ca\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-oauth-config\") pod \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\" (UID: \"fbfa5dde-3d55-459d-9eb5-250919bf4bc8\") " Sep 18 20:11:16.763733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763661 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:11:16.763733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763673 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:11:16.763733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763685 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-config" (OuterVolumeSpecName: "console-config") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:11:16.763862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.763730 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-service-ca" (OuterVolumeSpecName: "service-ca") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:11:16.765444 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.765420 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:11:16.765806 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.765788 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:11:16.765853 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.765802 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-kube-api-access-w547p" (OuterVolumeSpecName: "kube-api-access-w547p") pod "fbfa5dde-3d55-459d-9eb5-250919bf4bc8" (UID: "fbfa5dde-3d55-459d-9eb5-250919bf4bc8"). InnerVolumeSpecName "kube-api-access-w547p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:11:16.864843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864805 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:16.864843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864839 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-oauth-config\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:16.864843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864854 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w547p\" (UniqueName: \"kubernetes.io/projected/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-kube-api-access-w547p\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:16.865096 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864866 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-serving-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:16.865096 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864879 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-trusted-ca-bundle\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:16.865096 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864889 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-console-config\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:16.865096 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:16.864900 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbfa5dde-3d55-459d-9eb5-250919bf4bc8-oauth-serving-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:17.554098 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.554011 2559 generic.go:356] "Generic (PLEG): container finished" podID="fbfa5dde-3d55-459d-9eb5-250919bf4bc8" containerID="ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5" exitCode=0 Sep 18 20:11:17.554098 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.554050 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dfcc5c9d-kdwc6" Sep 18 20:11:17.554596 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.554099 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dfcc5c9d-kdwc6" event={"ID":"fbfa5dde-3d55-459d-9eb5-250919bf4bc8","Type":"ContainerDied","Data":"ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5"} Sep 18 20:11:17.554596 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.554149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dfcc5c9d-kdwc6" event={"ID":"fbfa5dde-3d55-459d-9eb5-250919bf4bc8","Type":"ContainerDied","Data":"4ea00a5a7a93c668f5b641d4bfd7524403aebded076a8ccb7fb807d306f59994"} Sep 18 20:11:17.554596 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.554170 2559 scope.go:117] "RemoveContainer" containerID="ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5" Sep 18 20:11:17.562611 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.562595 2559 scope.go:117] "RemoveContainer" containerID="ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5" Sep 18 20:11:17.562869 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:11:17.562848 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5\": container with ID starting with ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5 not found: ID does not exist" containerID="ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5" Sep 18 20:11:17.562931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.562877 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5"} err="failed to get container status \"ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5\": rpc error: code = NotFound desc = could not find container \"ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5\": container with ID starting with ae5478ab7f670332e3b5f493613259770941856c5e9494f5f6b76cfbf8fe6aa5 not found: ID does not exist" Sep 18 20:11:17.570849 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.570824 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75dfcc5c9d-kdwc6"] Sep 18 20:11:17.579970 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:17.575835 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75dfcc5c9d-kdwc6"] Sep 18 20:11:18.509126 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:18.509093 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wbbj2" Sep 18 20:11:19.183150 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:19.183114 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbfa5dde-3d55-459d-9eb5-250919bf4bc8" path="/var/lib/kubelet/pods/fbfa5dde-3d55-459d-9eb5-250919bf4bc8/volumes" Sep 18 20:11:23.117842 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:23.117802 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:11:23.118875 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:23.118852 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:11:36.582302 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.582207 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:36.584643 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.582556 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbfa5dde-3d55-459d-9eb5-250919bf4bc8" containerName="console" Sep 18 20:11:36.584643 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.582568 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbfa5dde-3d55-459d-9eb5-250919bf4bc8" containerName="console" Sep 18 20:11:36.584643 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.582625 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbfa5dde-3d55-459d-9eb5-250919bf4bc8" containerName="console" Sep 18 20:11:36.585605 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.585587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.588335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.588300 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 18 20:11:36.588335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.588328 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-75d4r\"" Sep 18 20:11:36.593055 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.593031 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:36.672610 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.672546 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:36.725814 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.725768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldz6f\" (UniqueName: \"kubernetes.io/projected/6b6a1db1-f906-467f-a861-6f778229cff3-kube-api-access-ldz6f\") pod \"limitador-limitador-8588749977-2gv5c\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.726016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.725831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6a1db1-f906-467f-a861-6f778229cff3-config-file\") pod \"limitador-limitador-8588749977-2gv5c\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.826568 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.826528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6a1db1-f906-467f-a861-6f778229cff3-config-file\") pod \"limitador-limitador-8588749977-2gv5c\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.826758 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.826640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldz6f\" (UniqueName: \"kubernetes.io/projected/6b6a1db1-f906-467f-a861-6f778229cff3-kube-api-access-ldz6f\") pod \"limitador-limitador-8588749977-2gv5c\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.827178 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.827157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6a1db1-f906-467f-a861-6f778229cff3-config-file\") pod \"limitador-limitador-8588749977-2gv5c\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.834655 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.834591 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldz6f\" (UniqueName: \"kubernetes.io/projected/6b6a1db1-f906-467f-a861-6f778229cff3-kube-api-access-ldz6f\") pod \"limitador-limitador-8588749977-2gv5c\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:36.897232 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:36.897188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:37.167381 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:37.167331 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:37.170325 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:11:37.170296 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6a1db1_f906_467f_a861_6f778229cff3.slice/crio-eef1507676ae7f8064e9595c9a7e857d6de2f57eb74f112514cfc97a5a384b78 WatchSource:0}: Error finding container eef1507676ae7f8064e9595c9a7e857d6de2f57eb74f112514cfc97a5a384b78: Status 404 returned error can't find the container with id eef1507676ae7f8064e9595c9a7e857d6de2f57eb74f112514cfc97a5a384b78 Sep 18 20:11:37.652211 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:37.652175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" event={"ID":"6b6a1db1-f906-467f-a861-6f778229cff3","Type":"ContainerStarted","Data":"eef1507676ae7f8064e9595c9a7e857d6de2f57eb74f112514cfc97a5a384b78"} Sep 18 20:11:38.658519 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:38.658482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" event={"ID":"6b6a1db1-f906-467f-a861-6f778229cff3","Type":"ContainerStarted","Data":"fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738"} Sep 18 20:11:38.658954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:38.658588 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:38.674645 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:38.674585 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" podStartSLOduration=1.686013656 podStartE2EDuration="2.674564852s" totalImagesPullingTime="988.551196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:38.672549311 +0000 UTC m=+617.722909761" watchObservedRunningTime="2026-09-18 20:11:38.674564852 +0000 UTC m=+617.724925300" Sep 18 20:11:49.664866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:49.664833 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:52.016895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.016860 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:52.017399 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.017045 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" podUID="6b6a1db1-f906-467f-a861-6f778229cff3" containerName="limitador" containerID="cri-o://fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738" gracePeriod=30 Sep 18 20:11:52.556292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.556243 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:52.651793 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.651759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldz6f\" (UniqueName: \"kubernetes.io/projected/6b6a1db1-f906-467f-a861-6f778229cff3-kube-api-access-ldz6f\") pod \"6b6a1db1-f906-467f-a861-6f778229cff3\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " Sep 18 20:11:52.651998 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.651811 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6a1db1-f906-467f-a861-6f778229cff3-config-file\") pod \"6b6a1db1-f906-467f-a861-6f778229cff3\" (UID: \"6b6a1db1-f906-467f-a861-6f778229cff3\") " Sep 18 20:11:52.652235 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.652205 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6a1db1-f906-467f-a861-6f778229cff3-config-file" (OuterVolumeSpecName: "config-file") pod "6b6a1db1-f906-467f-a861-6f778229cff3" (UID: "6b6a1db1-f906-467f-a861-6f778229cff3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:11:52.653872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.653853 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6a1db1-f906-467f-a861-6f778229cff3-kube-api-access-ldz6f" (OuterVolumeSpecName: "kube-api-access-ldz6f") pod "6b6a1db1-f906-467f-a861-6f778229cff3" (UID: "6b6a1db1-f906-467f-a861-6f778229cff3"). InnerVolumeSpecName "kube-api-access-ldz6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:11:52.725300 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.725247 2559 generic.go:356] "Generic (PLEG): container finished" podID="6b6a1db1-f906-467f-a861-6f778229cff3" containerID="fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738" exitCode=0 Sep 18 20:11:52.725300 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.725306 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" Sep 18 20:11:52.725591 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.725329 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" event={"ID":"6b6a1db1-f906-467f-a861-6f778229cff3","Type":"ContainerDied","Data":"fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738"} Sep 18 20:11:52.725591 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.725393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2gv5c" event={"ID":"6b6a1db1-f906-467f-a861-6f778229cff3","Type":"ContainerDied","Data":"eef1507676ae7f8064e9595c9a7e857d6de2f57eb74f112514cfc97a5a384b78"} Sep 18 20:11:52.725591 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.725426 2559 scope.go:117] "RemoveContainer" containerID="fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738" Sep 18 20:11:52.733861 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.733842 2559 scope.go:117] "RemoveContainer" containerID="fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738" Sep 18 20:11:52.734169 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:11:52.734151 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738\": container with ID starting with fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738 not found: ID does not exist" containerID="fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738" Sep 18 20:11:52.734215 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.734180 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738"} err="failed to get container status \"fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738\": rpc error: code = NotFound desc = could not find container \"fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738\": container with ID starting with fbccf051c019fe47456f450f535f5cde36fbd5cdde9df37da49f11c299211738 not found: ID does not exist" Sep 18 20:11:52.746610 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.746580 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:52.751875 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.751852 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2gv5c"] Sep 18 20:11:52.752641 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.752625 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldz6f\" (UniqueName: \"kubernetes.io/projected/6b6a1db1-f906-467f-a861-6f778229cff3-kube-api-access-ldz6f\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:52.752693 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:52.752644 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6a1db1-f906-467f-a861-6f778229cff3-config-file\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:11:53.183374 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:53.183338 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6a1db1-f906-467f-a861-6f778229cff3" path="/var/lib/kubelet/pods/6b6a1db1-f906-467f-a861-6f778229cff3/volumes" Sep 18 20:11:58.242809 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.242772 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s"] Sep 18 20:11:58.243181 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.243108 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6a1db1-f906-467f-a861-6f778229cff3" containerName="limitador" Sep 18 20:11:58.243181 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.243119 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6a1db1-f906-467f-a861-6f778229cff3" containerName="limitador" Sep 18 20:11:58.243181 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.243180 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6a1db1-f906-467f-a861-6f778229cff3" containerName="limitador" Sep 18 20:11:58.247649 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.247625 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.250830 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.250806 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 18 20:11:58.250973 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.250856 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 20:11:58.250973 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.250872 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 20:11:58.252066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.252044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-z7gzn\"" Sep 18 20:11:58.255688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.255665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s"] Sep 18 20:11:58.302950 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.302912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f286b7e9-ae5d-4006-a74d-db4fd564510c-data\") pod \"postgres-69fb4594cf-c2z6s\" (UID: \"f286b7e9-ae5d-4006-a74d-db4fd564510c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.303132 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.302982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66bl\" (UniqueName: \"kubernetes.io/projected/f286b7e9-ae5d-4006-a74d-db4fd564510c-kube-api-access-z66bl\") pod \"postgres-69fb4594cf-c2z6s\" (UID: \"f286b7e9-ae5d-4006-a74d-db4fd564510c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.404009 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.403968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66bl\" (UniqueName: \"kubernetes.io/projected/f286b7e9-ae5d-4006-a74d-db4fd564510c-kube-api-access-z66bl\") pod \"postgres-69fb4594cf-c2z6s\" (UID: \"f286b7e9-ae5d-4006-a74d-db4fd564510c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.404198 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.404035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f286b7e9-ae5d-4006-a74d-db4fd564510c-data\") pod \"postgres-69fb4594cf-c2z6s\" (UID: \"f286b7e9-ae5d-4006-a74d-db4fd564510c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.404449 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.404431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f286b7e9-ae5d-4006-a74d-db4fd564510c-data\") pod \"postgres-69fb4594cf-c2z6s\" (UID: \"f286b7e9-ae5d-4006-a74d-db4fd564510c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.413048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.413016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66bl\" (UniqueName: \"kubernetes.io/projected/f286b7e9-ae5d-4006-a74d-db4fd564510c-kube-api-access-z66bl\") pod \"postgres-69fb4594cf-c2z6s\" (UID: \"f286b7e9-ae5d-4006-a74d-db4fd564510c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.558243 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.558148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:11:58.829861 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:58.829834 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s"] Sep 18 20:11:58.832601 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:11:58.832567 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf286b7e9_ae5d_4006_a74d_db4fd564510c.slice/crio-1173805bbd914eb0d1ebaa571f186c0039c480da9141d34e407edef23a0be279 WatchSource:0}: Error finding container 1173805bbd914eb0d1ebaa571f186c0039c480da9141d34e407edef23a0be279: Status 404 returned error can't find the container with id 1173805bbd914eb0d1ebaa571f186c0039c480da9141d34e407edef23a0be279 Sep 18 20:11:59.764859 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:11:59.764817 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" event={"ID":"f286b7e9-ae5d-4006-a74d-db4fd564510c","Type":"ContainerStarted","Data":"1173805bbd914eb0d1ebaa571f186c0039c480da9141d34e407edef23a0be279"} Sep 18 20:12:03.793990 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:03.793956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" event={"ID":"f286b7e9-ae5d-4006-a74d-db4fd564510c","Type":"ContainerStarted","Data":"a34676f4b54a5807710bfb4ee7caca4bcba6e0be86840a2fc0c3203e17ca62ad"} Sep 18 20:12:03.794493 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:03.794070 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:12:03.812712 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:03.812641 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" podStartSLOduration=0.919466635 podStartE2EDuration="5.812617932s" totalImagesPullingTime="4.893151297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:03.81144588 +0000 UTC m=+642.861806331" watchObservedRunningTime="2026-09-18 20:12:03.812617932 +0000 UTC m=+642.862978380" Sep 18 20:12:09.827266 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:09.827234 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c2z6s" Sep 18 20:12:17.480368 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.480337 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-dmrbn"] Sep 18 20:12:17.496511 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.496482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.507760 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.507738 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 20:12:17.507760 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.507757 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 20:12:17.507915 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.507794 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 20:12:17.507915 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.507868 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h6cq7\"" Sep 18 20:12:17.520891 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.520859 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-dmrbn"] Sep 18 20:12:17.577738 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.577703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r95ww\" (UniqueName: \"kubernetes.io/projected/19e1e82c-19e0-424c-aabd-2521cc55d390-kube-api-access-r95ww\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.577936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.577774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-metrics-certs\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.577936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.577822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-webhook-cert\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.635248 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.635217 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fb768dd77-rgj8p"] Sep 18 20:12:17.637669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.637644 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.648173 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.648147 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-rgj8p"] Sep 18 20:12:17.678519 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.678479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-webhook-cert\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.678699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.678544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r95ww\" (UniqueName: \"kubernetes.io/projected/19e1e82c-19e0-424c-aabd-2521cc55d390-kube-api-access-r95ww\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.678699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.678586 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-metrics-certs\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.681093 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.681069 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-metrics-certs\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.681222 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.681201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-webhook-cert\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.686624 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.686602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r95ww\" (UniqueName: \"kubernetes.io/projected/19e1e82c-19e0-424c-aabd-2521cc55d390-kube-api-access-r95ww\") pod \"maas-controller-75dc6cbdf-dmrbn\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.779959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.779855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntlv\" (UniqueName: \"kubernetes.io/projected/485c8eec-2db1-41b5-a46f-699cb9ee70f7-kube-api-access-fntlv\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.779959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.779902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-webhook-cert\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.779959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.779918 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-metrics-certs\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.806856 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.806821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:17.881779 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.881737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntlv\" (UniqueName: \"kubernetes.io/projected/485c8eec-2db1-41b5-a46f-699cb9ee70f7-kube-api-access-fntlv\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.881955 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.881793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-webhook-cert\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.881955 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.881821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-metrics-certs\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.884498 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.884469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-webhook-cert\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.884611 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.884473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-metrics-certs\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.889542 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.889518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntlv\" (UniqueName: \"kubernetes.io/projected/485c8eec-2db1-41b5-a46f-699cb9ee70f7-kube-api-access-fntlv\") pod \"maas-controller-fb768dd77-rgj8p\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:17.947868 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:17.947822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:18.075235 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:18.075203 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-dmrbn"] Sep 18 20:12:18.077600 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:12:18.077576 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e1e82c_19e0_424c_aabd_2521cc55d390.slice/crio-2ff055c16153462a791a4eaa2c7f1bee260af228f12837e5c51cc9677225c8f5 WatchSource:0}: Error finding container 2ff055c16153462a791a4eaa2c7f1bee260af228f12837e5c51cc9677225c8f5: Status 404 returned error can't find the container with id 2ff055c16153462a791a4eaa2c7f1bee260af228f12837e5c51cc9677225c8f5 Sep 18 20:12:18.216765 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:18.216731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-rgj8p"] Sep 18 20:12:18.219511 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:12:18.219485 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485c8eec_2db1_41b5_a46f_699cb9ee70f7.slice/crio-3d010dcbdfafbfc0c9abf58ab6a78508a1258a4cdb6c95a7d7ba85dfc605971a WatchSource:0}: Error finding container 3d010dcbdfafbfc0c9abf58ab6a78508a1258a4cdb6c95a7d7ba85dfc605971a: Status 404 returned error can't find the container with id 3d010dcbdfafbfc0c9abf58ab6a78508a1258a4cdb6c95a7d7ba85dfc605971a Sep 18 20:12:18.878101 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:18.878059 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-rgj8p" event={"ID":"485c8eec-2db1-41b5-a46f-699cb9ee70f7","Type":"ContainerStarted","Data":"3d010dcbdfafbfc0c9abf58ab6a78508a1258a4cdb6c95a7d7ba85dfc605971a"} Sep 18 20:12:18.880267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:18.880240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" event={"ID":"19e1e82c-19e0-424c-aabd-2521cc55d390","Type":"ContainerStarted","Data":"2ff055c16153462a791a4eaa2c7f1bee260af228f12837e5c51cc9677225c8f5"} Sep 18 20:12:21.898181 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:21.898142 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-rgj8p" event={"ID":"485c8eec-2db1-41b5-a46f-699cb9ee70f7","Type":"ContainerStarted","Data":"4c00ddf569f1cd462ff799638e4990a6cbd9cc5bee10d3f72dfa6066a43716f6"} Sep 18 20:12:21.898680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:21.898260 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:21.899927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:21.899900 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" event={"ID":"19e1e82c-19e0-424c-aabd-2521cc55d390","Type":"ContainerStarted","Data":"7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399"} Sep 18 20:12:21.900029 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:21.899958 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:21.916702 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:21.916648 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fb768dd77-rgj8p" podStartSLOduration=2.291888161 podStartE2EDuration="4.91663214s" totalImagesPullingTime="2.624743979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:21.914111599 +0000 UTC m=+660.964472046" watchObservedRunningTime="2026-09-18 20:12:21.91663214 +0000 UTC m=+660.966992587" Sep 18 20:12:21.931724 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:21.931667 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" podStartSLOduration=2.174599676 podStartE2EDuration="4.931651478s" totalImagesPullingTime="2.757051802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:21.930193021 +0000 UTC m=+660.980553483" watchObservedRunningTime="2026-09-18 20:12:21.931651478 +0000 UTC m=+660.982011925" Sep 18 20:12:32.906835 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:32.906804 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:32.907247 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:32.906903 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:12:32.973593 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:32.973531 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-dmrbn"] Sep 18 20:12:32.973851 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:32.973777 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" podUID="19e1e82c-19e0-424c-aabd-2521cc55d390" containerName="manager" containerID="cri-o://7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399" gracePeriod=30 Sep 18 20:12:33.227878 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.227851 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:33.319676 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.319648 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-metrics-certs\") pod \"19e1e82c-19e0-424c-aabd-2521cc55d390\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " Sep 18 20:12:33.319879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.319693 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r95ww\" (UniqueName: \"kubernetes.io/projected/19e1e82c-19e0-424c-aabd-2521cc55d390-kube-api-access-r95ww\") pod \"19e1e82c-19e0-424c-aabd-2521cc55d390\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " Sep 18 20:12:33.319879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.319727 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-webhook-cert\") pod \"19e1e82c-19e0-424c-aabd-2521cc55d390\" (UID: \"19e1e82c-19e0-424c-aabd-2521cc55d390\") " Sep 18 20:12:33.321989 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.321957 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "19e1e82c-19e0-424c-aabd-2521cc55d390" (UID: "19e1e82c-19e0-424c-aabd-2521cc55d390"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:12:33.322118 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.321991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "19e1e82c-19e0-424c-aabd-2521cc55d390" (UID: "19e1e82c-19e0-424c-aabd-2521cc55d390"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:12:33.322118 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.322002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e1e82c-19e0-424c-aabd-2521cc55d390-kube-api-access-r95ww" (OuterVolumeSpecName: "kube-api-access-r95ww") pod "19e1e82c-19e0-424c-aabd-2521cc55d390" (UID: "19e1e82c-19e0-424c-aabd-2521cc55d390"). InnerVolumeSpecName "kube-api-access-r95ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:12:33.420539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.420503 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-webhook-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:12:33.420539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.420536 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e1e82c-19e0-424c-aabd-2521cc55d390-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:12:33.420539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.420545 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r95ww\" (UniqueName: \"kubernetes.io/projected/19e1e82c-19e0-424c-aabd-2521cc55d390-kube-api-access-r95ww\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:12:33.957044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.957005 2559 generic.go:356] "Generic (PLEG): container finished" podID="19e1e82c-19e0-424c-aabd-2521cc55d390" containerID="7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399" exitCode=0 Sep 18 20:12:33.957472 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.957050 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" Sep 18 20:12:33.957472 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.957092 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" event={"ID":"19e1e82c-19e0-424c-aabd-2521cc55d390","Type":"ContainerDied","Data":"7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399"} Sep 18 20:12:33.957472 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.957153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-dmrbn" event={"ID":"19e1e82c-19e0-424c-aabd-2521cc55d390","Type":"ContainerDied","Data":"2ff055c16153462a791a4eaa2c7f1bee260af228f12837e5c51cc9677225c8f5"} Sep 18 20:12:33.957472 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.957201 2559 scope.go:117] "RemoveContainer" containerID="7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399" Sep 18 20:12:33.966537 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.966520 2559 scope.go:117] "RemoveContainer" containerID="7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399" Sep 18 20:12:33.966832 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:12:33.966812 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399\": container with ID starting with 7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399 not found: ID does not exist" containerID="7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399" Sep 18 20:12:33.966880 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.966842 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399"} err="failed to get container status \"7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399\": rpc error: code = NotFound desc = could not find container \"7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399\": container with ID starting with 7e6c25ee7fe8c5ccc2ad4e94ad7a884b464066da7f8482c8e93463daeaff8399 not found: ID does not exist" Sep 18 20:12:33.978715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.978685 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-dmrbn"] Sep 18 20:12:33.981936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:33.981911 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-dmrbn"] Sep 18 20:12:35.183613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:35.183569 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e1e82c-19e0-424c-aabd-2521cc55d390" path="/var/lib/kubelet/pods/19e1e82c-19e0-424c-aabd-2521cc55d390/volumes" Sep 18 20:12:55.160547 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.160512 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52"] Sep 18 20:12:55.161025 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.160878 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e1e82c-19e0-424c-aabd-2521cc55d390" containerName="manager" Sep 18 20:12:55.161025 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.160889 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e1e82c-19e0-424c-aabd-2521cc55d390" containerName="manager" Sep 18 20:12:55.161025 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.160964 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e1e82c-19e0-424c-aabd-2521cc55d390" containerName="manager" Sep 18 20:12:55.165471 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.165453 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.168372 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.168343 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 18 20:12:55.168372 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.168359 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fhqv4\"" Sep 18 20:12:55.168562 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.168360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 20:12:55.175311 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.175265 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52"] Sep 18 20:12:55.211230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.211194 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-openshift-service-ca\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.211415 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.211248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-maas-api-tls\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.211461 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.211427 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gnd\" (UniqueName: \"kubernetes.io/projected/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-kube-api-access-52gnd\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.211495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.211460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.311921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.311870 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gnd\" (UniqueName: \"kubernetes.io/projected/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-kube-api-access-52gnd\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.311921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.311925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.312187 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.311949 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-openshift-service-ca\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.312187 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.311994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-maas-api-tls\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.312187 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:12:55.312058 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 18 20:12:55.312187 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:12:55.312146 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs podName:ab6af800-ebb0-4a94-8174-7b2f83ea7b15 nodeName:}" failed. No retries permitted until 2026-09-18 20:12:55.812126282 +0000 UTC m=+694.862486715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs") pod "maas-api-59f6566bbc-x7m52" (UID: "ab6af800-ebb0-4a94-8174-7b2f83ea7b15") : secret "maas-api-metrics-tls" not found Sep 18 20:12:55.312752 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.312726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-openshift-service-ca\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.314556 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.314537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-maas-api-tls\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.320913 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.320888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gnd\" (UniqueName: \"kubernetes.io/projected/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-kube-api-access-52gnd\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.681457 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.681417 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2xm5d"] Sep 18 20:12:55.684737 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.684712 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.687665 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.687637 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dmsxp\"" Sep 18 20:12:55.687797 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.687671 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 18 20:12:55.694520 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.694486 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2xm5d"] Sep 18 20:12:55.815750 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.815714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2fs\" (UniqueName: \"kubernetes.io/projected/e3f48e3a-f44e-479d-bbf6-2ad60b406847-kube-api-access-zr2fs\") pod \"payload-processing-5b54b7fbb7-2xm5d\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.815956 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.815763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.815956 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.815842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f48e3a-f44e-479d-bbf6-2ad60b406847-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2xm5d\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.818485 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.818459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs\") pod \"maas-api-59f6566bbc-x7m52\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:55.917325 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.917258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2fs\" (UniqueName: \"kubernetes.io/projected/e3f48e3a-f44e-479d-bbf6-2ad60b406847-kube-api-access-zr2fs\") pod \"payload-processing-5b54b7fbb7-2xm5d\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.917505 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.917341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f48e3a-f44e-479d-bbf6-2ad60b406847-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2xm5d\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.918030 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.918011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f48e3a-f44e-479d-bbf6-2ad60b406847-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2xm5d\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.927660 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.927636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2fs\" (UniqueName: \"kubernetes.io/projected/e3f48e3a-f44e-479d-bbf6-2ad60b406847-kube-api-access-zr2fs\") pod \"payload-processing-5b54b7fbb7-2xm5d\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:55.996883 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:55.996790 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:56.077072 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:56.076975 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:56.268337 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:56.268311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2xm5d"] Sep 18 20:12:56.270905 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:12:56.270874 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f48e3a_f44e_479d_bbf6_2ad60b406847.slice/crio-ae4cf9cdb59ab35700909e4e8b863aeda7a309f2e2242df89b51ad5dc1649ad1 WatchSource:0}: Error finding container ae4cf9cdb59ab35700909e4e8b863aeda7a309f2e2242df89b51ad5dc1649ad1: Status 404 returned error can't find the container with id ae4cf9cdb59ab35700909e4e8b863aeda7a309f2e2242df89b51ad5dc1649ad1 Sep 18 20:12:56.353213 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:56.353188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52"] Sep 18 20:12:56.356416 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:12:56.356386 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6af800_ebb0_4a94_8174_7b2f83ea7b15.slice/crio-d60bc2ec41afea99b0f90c7d528e91b344384e90bbe5620af22069d469e5f016 WatchSource:0}: Error finding container d60bc2ec41afea99b0f90c7d528e91b344384e90bbe5620af22069d469e5f016: Status 404 returned error can't find the container with id d60bc2ec41afea99b0f90c7d528e91b344384e90bbe5620af22069d469e5f016 Sep 18 20:12:57.073715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:57.073635 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" event={"ID":"e3f48e3a-f44e-479d-bbf6-2ad60b406847","Type":"ContainerStarted","Data":"ae4cf9cdb59ab35700909e4e8b863aeda7a309f2e2242df89b51ad5dc1649ad1"} Sep 18 20:12:57.075378 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:57.075348 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" event={"ID":"ab6af800-ebb0-4a94-8174-7b2f83ea7b15","Type":"ContainerStarted","Data":"d60bc2ec41afea99b0f90c7d528e91b344384e90bbe5620af22069d469e5f016"} Sep 18 20:12:59.087710 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:59.087670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" event={"ID":"ab6af800-ebb0-4a94-8174-7b2f83ea7b15","Type":"ContainerStarted","Data":"9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c"} Sep 18 20:12:59.088191 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:59.087745 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:12:59.089424 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:59.089400 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" event={"ID":"e3f48e3a-f44e-479d-bbf6-2ad60b406847","Type":"ContainerStarted","Data":"4a5494955847b90b01f24e46cb0ea1bde991676127fa4df470fdc7e00534b3f8"} Sep 18 20:12:59.089549 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:59.089523 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:12:59.107790 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:59.107728 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" podStartSLOduration=1.50243946 podStartE2EDuration="4.107704785s" totalImagesPullingTime="2.605265325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:59.106817239 +0000 UTC m=+698.157177702" watchObservedRunningTime="2026-09-18 20:12:59.107704785 +0000 UTC m=+698.158065233" Sep 18 20:12:59.124024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:12:59.123967 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" podStartSLOduration=1.432171608 podStartE2EDuration="4.123950344s" totalImagesPullingTime="2.691778736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:59.122745087 +0000 UTC m=+698.173105536" watchObservedRunningTime="2026-09-18 20:12:59.123950344 +0000 UTC m=+698.174310792" Sep 18 20:13:05.101405 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:05.101325 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:13:10.097391 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:10.097354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:13:52.302213 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.302174 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg"] Sep 18 20:13:52.304584 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.304567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.308508 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.308483 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 20:13:52.308685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.308503 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 18 20:13:52.308685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.308556 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 20:13:52.308843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.308566 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m7b4l\"" Sep 18 20:13:52.314704 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.314674 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg"] Sep 18 20:13:52.425687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.425643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.425687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.425696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.425930 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.425714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.425930 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.425810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869e4bad-f7a6-459c-9a50-8fe45a479c21-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.425930 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.425866 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwxw\" (UniqueName: \"kubernetes.io/projected/869e4bad-f7a6-459c-9a50-8fe45a479c21-kube-api-access-thwxw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.426043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.425940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.526663 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.526628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.526872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.526680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.526872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.526809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.526872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.526850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.527040 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.526907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869e4bad-f7a6-459c-9a50-8fe45a479c21-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.527040 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.527014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.527143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.527059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwxw\" (UniqueName: \"kubernetes.io/projected/869e4bad-f7a6-459c-9a50-8fe45a479c21-kube-api-access-thwxw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.527143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.527080 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.527246 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.527204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.529024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.529004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869e4bad-f7a6-459c-9a50-8fe45a479c21-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.529390 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.529372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869e4bad-f7a6-459c-9a50-8fe45a479c21-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.535164 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.535131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwxw\" (UniqueName: \"kubernetes.io/projected/869e4bad-f7a6-459c-9a50-8fe45a479c21-kube-api-access-thwxw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg\" (UID: \"869e4bad-f7a6-459c-9a50-8fe45a479c21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.617714 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.617680 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:13:52.888018 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.887992 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg"] Sep 18 20:13:52.890391 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:13:52.890365 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869e4bad_f7a6_459c_9a50_8fe45a479c21.slice/crio-c837e95b4e48db0fe4f478444e989eb0606c8babe5221e517c83c3f32ece9ba3 WatchSource:0}: Error finding container c837e95b4e48db0fe4f478444e989eb0606c8babe5221e517c83c3f32ece9ba3: Status 404 returned error can't find the container with id c837e95b4e48db0fe4f478444e989eb0606c8babe5221e517c83c3f32ece9ba3 Sep 18 20:13:52.891803 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:52.891782 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:13:53.353092 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:53.353055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" event={"ID":"869e4bad-f7a6-459c-9a50-8fe45a479c21","Type":"ContainerStarted","Data":"c837e95b4e48db0fe4f478444e989eb0606c8babe5221e517c83c3f32ece9ba3"} Sep 18 20:13:54.360186 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:54.360153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" event={"ID":"869e4bad-f7a6-459c-9a50-8fe45a479c21","Type":"ContainerStarted","Data":"c0d57c927b27b4a4d88785ead1b0f221836d08c45b0a16d69400626c4d79824e"} Sep 18 20:13:54.377471 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:13:54.377416 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" podStartSLOduration=1.441017324 podStartE2EDuration="2.377399012s" totalImagesPullingTime="936.381688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:54.376079875 +0000 UTC m=+753.426440319" watchObservedRunningTime="2026-09-18 20:13:54.377399012 +0000 UTC m=+753.427759460" Sep 18 20:14:02.618250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:02.618202 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:14:02.618722 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:02.618311 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:14:02.630892 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:02.630862 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:14:03.413947 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:03.413914 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g7kqg" Sep 18 20:14:38.787616 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.787529 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz"] Sep 18 20:14:38.790347 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.790322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:38.793327 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.793266 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 18 20:14:38.804672 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.804631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz"] Sep 18 20:14:38.949024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.948982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:38.949024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.949031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:38.949265 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.949113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:38.949265 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.949152 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmsd\" (UniqueName: \"kubernetes.io/projected/954ac38d-260e-4c76-b71e-200ec1112b92-kube-api-access-tqmsd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:38.949410 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.949270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:38.949410 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:38.949323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954ac38d-260e-4c76-b71e-200ec1112b92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.050719 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.050610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.050719 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.050662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954ac38d-260e-4c76-b71e-200ec1112b92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.050719 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.050713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.051017 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.050738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.051017 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.050779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.051017 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.050803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmsd\" (UniqueName: \"kubernetes.io/projected/954ac38d-260e-4c76-b71e-200ec1112b92-kube-api-access-tqmsd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.051168 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.051095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.051168 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.051141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.051239 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.051191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.053071 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.053052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954ac38d-260e-4c76-b71e-200ec1112b92-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.053332 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.053315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954ac38d-260e-4c76-b71e-200ec1112b92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.059839 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.059811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmsd\" (UniqueName: \"kubernetes.io/projected/954ac38d-260e-4c76-b71e-200ec1112b92-kube-api-access-tqmsd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f28cz\" (UID: \"954ac38d-260e-4c76-b71e-200ec1112b92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.105522 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.105480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:39.377864 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.377835 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz"] Sep 18 20:14:39.379691 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:14:39.379656 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod954ac38d_260e_4c76_b71e_200ec1112b92.slice/crio-01e932671eb5a4e76ab0487e6a0dc628858f92580ef28c9a7c5316526299cbed WatchSource:0}: Error finding container 01e932671eb5a4e76ab0487e6a0dc628858f92580ef28c9a7c5316526299cbed: Status 404 returned error can't find the container with id 01e932671eb5a4e76ab0487e6a0dc628858f92580ef28c9a7c5316526299cbed Sep 18 20:14:39.571903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:39.571875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" event={"ID":"954ac38d-260e-4c76-b71e-200ec1112b92","Type":"ContainerStarted","Data":"01e932671eb5a4e76ab0487e6a0dc628858f92580ef28c9a7c5316526299cbed"} Sep 18 20:14:40.577960 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:40.577920 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" event={"ID":"954ac38d-260e-4c76-b71e-200ec1112b92","Type":"ContainerStarted","Data":"72a4b0aefb3d12541396fbe6f18c8d3d4a951d809ec3c1941a568690cca62ac5"} Sep 18 20:14:40.595586 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:40.595529 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" podStartSLOduration=2.434216159 podStartE2EDuration="2.595510885s" totalImagesPullingTime="161.294726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:40.594190184 +0000 UTC m=+799.644550632" watchObservedRunningTime="2026-09-18 20:14:40.595510885 +0000 UTC m=+799.645871332" Sep 18 20:14:49.105884 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:49.105844 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:49.106363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:49.105901 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:49.118802 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:49.118773 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:14:49.631845 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:14:49.631812 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f28cz" Sep 18 20:15:51.792521 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.792486 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fb768dd77-rgj8p"] Sep 18 20:15:51.793083 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.792713 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fb768dd77-rgj8p" podUID="485c8eec-2db1-41b5-a46f-699cb9ee70f7" containerName="manager" containerID="cri-o://4c00ddf569f1cd462ff799638e4990a6cbd9cc5bee10d3f72dfa6066a43716f6" gracePeriod=30 Sep 18 20:15:51.796520 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.796484 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2xm5d"] Sep 18 20:15:51.796740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.796714 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" podUID="e3f48e3a-f44e-479d-bbf6-2ad60b406847" containerName="payload-processing" containerID="cri-o://4a5494955847b90b01f24e46cb0ea1bde991676127fa4df470fdc7e00534b3f8" gracePeriod=30 Sep 18 20:15:51.923568 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.923528 2559 generic.go:356] "Generic (PLEG): container finished" podID="e3f48e3a-f44e-479d-bbf6-2ad60b406847" containerID="4a5494955847b90b01f24e46cb0ea1bde991676127fa4df470fdc7e00534b3f8" exitCode=0 Sep 18 20:15:51.923745 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.923622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" event={"ID":"e3f48e3a-f44e-479d-bbf6-2ad60b406847","Type":"ContainerDied","Data":"4a5494955847b90b01f24e46cb0ea1bde991676127fa4df470fdc7e00534b3f8"} Sep 18 20:15:51.925563 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.925534 2559 generic.go:356] "Generic (PLEG): container finished" podID="485c8eec-2db1-41b5-a46f-699cb9ee70f7" containerID="4c00ddf569f1cd462ff799638e4990a6cbd9cc5bee10d3f72dfa6066a43716f6" exitCode=0 Sep 18 20:15:51.925722 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:51.925589 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-rgj8p" event={"ID":"485c8eec-2db1-41b5-a46f-699cb9ee70f7","Type":"ContainerDied","Data":"4c00ddf569f1cd462ff799638e4990a6cbd9cc5bee10d3f72dfa6066a43716f6"} Sep 18 20:15:52.074836 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.074810 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:15:52.078169 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.078135 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:15:52.220870 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.220836 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-metrics-certs\") pod \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " Sep 18 20:15:52.221079 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.220886 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr2fs\" (UniqueName: \"kubernetes.io/projected/e3f48e3a-f44e-479d-bbf6-2ad60b406847-kube-api-access-zr2fs\") pod \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " Sep 18 20:15:52.221079 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.220908 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-webhook-cert\") pod \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " Sep 18 20:15:52.221079 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.220944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f48e3a-f44e-479d-bbf6-2ad60b406847-plugins-config-volume\") pod \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\" (UID: \"e3f48e3a-f44e-479d-bbf6-2ad60b406847\") " Sep 18 20:15:52.221079 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.220998 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fntlv\" (UniqueName: \"kubernetes.io/projected/485c8eec-2db1-41b5-a46f-699cb9ee70f7-kube-api-access-fntlv\") pod \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\" (UID: \"485c8eec-2db1-41b5-a46f-699cb9ee70f7\") " Sep 18 20:15:52.221377 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.221331 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f48e3a-f44e-479d-bbf6-2ad60b406847-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3f48e3a-f44e-479d-bbf6-2ad60b406847" (UID: "e3f48e3a-f44e-479d-bbf6-2ad60b406847"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:15:52.223162 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.223132 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485c8eec-2db1-41b5-a46f-699cb9ee70f7-kube-api-access-fntlv" (OuterVolumeSpecName: "kube-api-access-fntlv") pod "485c8eec-2db1-41b5-a46f-699cb9ee70f7" (UID: "485c8eec-2db1-41b5-a46f-699cb9ee70f7"). InnerVolumeSpecName "kube-api-access-fntlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:15:52.223296 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.223239 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f48e3a-f44e-479d-bbf6-2ad60b406847-kube-api-access-zr2fs" (OuterVolumeSpecName: "kube-api-access-zr2fs") pod "e3f48e3a-f44e-479d-bbf6-2ad60b406847" (UID: "e3f48e3a-f44e-479d-bbf6-2ad60b406847"). InnerVolumeSpecName "kube-api-access-zr2fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:15:52.223529 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.223504 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "485c8eec-2db1-41b5-a46f-699cb9ee70f7" (UID: "485c8eec-2db1-41b5-a46f-699cb9ee70f7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:15:52.223529 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.223518 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "485c8eec-2db1-41b5-a46f-699cb9ee70f7" (UID: "485c8eec-2db1-41b5-a46f-699cb9ee70f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:15:52.322369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.322234 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f48e3a-f44e-479d-bbf6-2ad60b406847-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:15:52.322369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.322303 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fntlv\" (UniqueName: \"kubernetes.io/projected/485c8eec-2db1-41b5-a46f-699cb9ee70f7-kube-api-access-fntlv\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:15:52.322369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.322319 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:15:52.322369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.322333 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2fs\" (UniqueName: \"kubernetes.io/projected/e3f48e3a-f44e-479d-bbf6-2ad60b406847-kube-api-access-zr2fs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:15:52.322369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.322348 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485c8eec-2db1-41b5-a46f-699cb9ee70f7-webhook-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:15:52.813516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813486 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fb768dd77-gpl6w"] Sep 18 20:15:52.813855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813847 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485c8eec-2db1-41b5-a46f-699cb9ee70f7" containerName="manager" Sep 18 20:15:52.813895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813859 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="485c8eec-2db1-41b5-a46f-699cb9ee70f7" containerName="manager" Sep 18 20:15:52.813895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813866 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f48e3a-f44e-479d-bbf6-2ad60b406847" containerName="payload-processing" Sep 18 20:15:52.813895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813872 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f48e3a-f44e-479d-bbf6-2ad60b406847" containerName="payload-processing" Sep 18 20:15:52.813981 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813922 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f48e3a-f44e-479d-bbf6-2ad60b406847" containerName="payload-processing" Sep 18 20:15:52.813981 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.813933 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="485c8eec-2db1-41b5-a46f-699cb9ee70f7" containerName="manager" Sep 18 20:15:52.816373 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.816356 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:52.826842 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.826816 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-gpl6w"] Sep 18 20:15:52.927777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.927747 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-metrics-certs\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:52.927984 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.927801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-webhook-cert\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:52.927984 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.927850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfm8\" (UniqueName: \"kubernetes.io/projected/57af45fb-aa04-4521-8273-40271aa49356-kube-api-access-wsfm8\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:52.932044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.932017 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" Sep 18 20:15:52.932203 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.932055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2xm5d" event={"ID":"e3f48e3a-f44e-479d-bbf6-2ad60b406847","Type":"ContainerDied","Data":"ae4cf9cdb59ab35700909e4e8b863aeda7a309f2e2242df89b51ad5dc1649ad1"} Sep 18 20:15:52.932203 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.932129 2559 scope.go:117] "RemoveContainer" containerID="4a5494955847b90b01f24e46cb0ea1bde991676127fa4df470fdc7e00534b3f8" Sep 18 20:15:52.933497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.933443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-rgj8p" event={"ID":"485c8eec-2db1-41b5-a46f-699cb9ee70f7","Type":"ContainerDied","Data":"3d010dcbdfafbfc0c9abf58ab6a78508a1258a4cdb6c95a7d7ba85dfc605971a"} Sep 18 20:15:52.933497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.933473 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-rgj8p" Sep 18 20:15:52.942021 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.942008 2559 scope.go:117] "RemoveContainer" containerID="4c00ddf569f1cd462ff799638e4990a6cbd9cc5bee10d3f72dfa6066a43716f6" Sep 18 20:15:52.954134 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.954105 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2xm5d"] Sep 18 20:15:52.959674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.959648 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2xm5d"] Sep 18 20:15:52.972229 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.972203 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fb768dd77-rgj8p"] Sep 18 20:15:52.976737 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:52.976713 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fb768dd77-rgj8p"] Sep 18 20:15:53.029131 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.029097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-webhook-cert\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.029314 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.029146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfm8\" (UniqueName: \"kubernetes.io/projected/57af45fb-aa04-4521-8273-40271aa49356-kube-api-access-wsfm8\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.029314 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.029191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-metrics-certs\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.031662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.031638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-metrics-certs\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.031767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.031689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-webhook-cert\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.037253 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.037235 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfm8\" (UniqueName: \"kubernetes.io/projected/57af45fb-aa04-4521-8273-40271aa49356-kube-api-access-wsfm8\") pod \"maas-controller-fb768dd77-gpl6w\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.127110 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.127060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.188326 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.188300 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485c8eec-2db1-41b5-a46f-699cb9ee70f7" path="/var/lib/kubelet/pods/485c8eec-2db1-41b5-a46f-699cb9ee70f7/volumes" Sep 18 20:15:53.189082 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.189050 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f48e3a-f44e-479d-bbf6-2ad60b406847" path="/var/lib/kubelet/pods/e3f48e3a-f44e-479d-bbf6-2ad60b406847/volumes" Sep 18 20:15:53.397134 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.397048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-gpl6w"] Sep 18 20:15:53.400233 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:15:53.400202 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57af45fb_aa04_4521_8273_40271aa49356.slice/crio-cf48a6bb75504a81013c337b3929def0724a3118ba75181f2eb7605622dccf5f WatchSource:0}: Error finding container cf48a6bb75504a81013c337b3929def0724a3118ba75181f2eb7605622dccf5f: Status 404 returned error can't find the container with id cf48a6bb75504a81013c337b3929def0724a3118ba75181f2eb7605622dccf5f Sep 18 20:15:53.940651 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.940613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-gpl6w" event={"ID":"57af45fb-aa04-4521-8273-40271aa49356","Type":"ContainerStarted","Data":"72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f"} Sep 18 20:15:53.941108 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.940668 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-gpl6w" event={"ID":"57af45fb-aa04-4521-8273-40271aa49356","Type":"ContainerStarted","Data":"cf48a6bb75504a81013c337b3929def0724a3118ba75181f2eb7605622dccf5f"} Sep 18 20:15:53.941108 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.940711 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:15:53.958463 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:15:53.958409 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fb768dd77-gpl6w" podStartSLOduration=1.6108145839999999 podStartE2EDuration="1.958387288s" totalImagesPullingTime="347.572704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:53.957146279 +0000 UTC m=+873.007506728" watchObservedRunningTime="2026-09-18 20:15:53.958387288 +0000 UTC m=+873.008747737" Sep 18 20:16:04.948879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:04.948799 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:16:05.200090 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.200007 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6db44cc5dd-8scfh"] Sep 18 20:16:05.202390 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.202373 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:05.205186 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.205162 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Sep 18 20:16:05.205347 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.205240 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-7mx6s\"" Sep 18 20:16:05.210982 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.210955 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6db44cc5dd-8scfh"] Sep 18 20:16:05.343464 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.343424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvhk\" (UniqueName: \"kubernetes.io/projected/857cf02a-c311-4a2d-adb2-b17227f3f5a8-kube-api-access-bsvhk\") pod \"ai-gateway-controller-6db44cc5dd-8scfh\" (UID: \"857cf02a-c311-4a2d-adb2-b17227f3f5a8\") " pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:05.444329 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.444293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvhk\" (UniqueName: \"kubernetes.io/projected/857cf02a-c311-4a2d-adb2-b17227f3f5a8-kube-api-access-bsvhk\") pod \"ai-gateway-controller-6db44cc5dd-8scfh\" (UID: \"857cf02a-c311-4a2d-adb2-b17227f3f5a8\") " pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:05.453262 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.453193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvhk\" (UniqueName: \"kubernetes.io/projected/857cf02a-c311-4a2d-adb2-b17227f3f5a8-kube-api-access-bsvhk\") pod \"ai-gateway-controller-6db44cc5dd-8scfh\" (UID: \"857cf02a-c311-4a2d-adb2-b17227f3f5a8\") " pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:05.514536 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.514479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:05.783808 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:05.783727 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6db44cc5dd-8scfh"] Sep 18 20:16:05.786805 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:05.786775 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857cf02a_c311_4a2d_adb2_b17227f3f5a8.slice/crio-c6910a2122935dde4b733069473106a3650552d9a3e14a0be6d04019310c1d72 WatchSource:0}: Error finding container c6910a2122935dde4b733069473106a3650552d9a3e14a0be6d04019310c1d72: Status 404 returned error can't find the container with id c6910a2122935dde4b733069473106a3650552d9a3e14a0be6d04019310c1d72 Sep 18 20:16:06.003296 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:06.003238 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" event={"ID":"857cf02a-c311-4a2d-adb2-b17227f3f5a8","Type":"ContainerStarted","Data":"c6910a2122935dde4b733069473106a3650552d9a3e14a0be6d04019310c1d72"} Sep 18 20:16:08.014803 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:08.014700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" event={"ID":"857cf02a-c311-4a2d-adb2-b17227f3f5a8","Type":"ContainerStarted","Data":"dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0"} Sep 18 20:16:08.014803 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:08.014791 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:08.031830 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:08.031778 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" podStartSLOduration=1.104210816 podStartE2EDuration="3.03176084s" totalImagesPullingTime="1.927550024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:08.029988526 +0000 UTC m=+887.080348975" watchObservedRunningTime="2026-09-18 20:16:08.03176084 +0000 UTC m=+887.082121287" Sep 18 20:16:13.174059 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.174018 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-7wlzn"] Sep 18 20:16:13.177078 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.177052 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.179594 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.179565 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bhjtf\"" Sep 18 20:16:13.180755 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.180729 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 18 20:16:13.186380 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.186352 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-7wlzn"] Sep 18 20:16:13.215899 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.215863 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxw2\" (UniqueName: \"kubernetes.io/projected/903c000c-3e8a-4eea-8ef7-5d3d52496350-kube-api-access-9xxw2\") pod \"payload-processing-85fd65c94c-7wlzn\" (UID: \"903c000c-3e8a-4eea-8ef7-5d3d52496350\") " pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.216059 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.215931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903c000c-3e8a-4eea-8ef7-5d3d52496350-plugins-config-volume\") pod \"payload-processing-85fd65c94c-7wlzn\" (UID: \"903c000c-3e8a-4eea-8ef7-5d3d52496350\") " pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.317257 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.317223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxw2\" (UniqueName: \"kubernetes.io/projected/903c000c-3e8a-4eea-8ef7-5d3d52496350-kube-api-access-9xxw2\") pod \"payload-processing-85fd65c94c-7wlzn\" (UID: \"903c000c-3e8a-4eea-8ef7-5d3d52496350\") " pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.317445 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.317328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903c000c-3e8a-4eea-8ef7-5d3d52496350-plugins-config-volume\") pod \"payload-processing-85fd65c94c-7wlzn\" (UID: \"903c000c-3e8a-4eea-8ef7-5d3d52496350\") " pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.318124 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.318105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903c000c-3e8a-4eea-8ef7-5d3d52496350-plugins-config-volume\") pod \"payload-processing-85fd65c94c-7wlzn\" (UID: \"903c000c-3e8a-4eea-8ef7-5d3d52496350\") " pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.325804 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.325780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxw2\" (UniqueName: \"kubernetes.io/projected/903c000c-3e8a-4eea-8ef7-5d3d52496350-kube-api-access-9xxw2\") pod \"payload-processing-85fd65c94c-7wlzn\" (UID: \"903c000c-3e8a-4eea-8ef7-5d3d52496350\") " pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.490014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.489926 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:13.760047 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:13.760019 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-7wlzn"] Sep 18 20:16:13.762249 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:13.762213 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903c000c_3e8a_4eea_8ef7_5d3d52496350.slice/crio-05b422e938cbb067d2f6afe2933f7d42e0c580059e37e9b8ad6c6c9625b102fc WatchSource:0}: Error finding container 05b422e938cbb067d2f6afe2933f7d42e0c580059e37e9b8ad6c6c9625b102fc: Status 404 returned error can't find the container with id 05b422e938cbb067d2f6afe2933f7d42e0c580059e37e9b8ad6c6c9625b102fc Sep 18 20:16:14.046966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:14.046874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" event={"ID":"903c000c-3e8a-4eea-8ef7-5d3d52496350","Type":"ContainerStarted","Data":"05b422e938cbb067d2f6afe2933f7d42e0c580059e37e9b8ad6c6c9625b102fc"} Sep 18 20:16:15.056340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:15.056300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" event={"ID":"903c000c-3e8a-4eea-8ef7-5d3d52496350","Type":"ContainerStarted","Data":"4c7f81341a545783456ece81136314fcdff84acb36fbd9309cec0cebf34be9f0"} Sep 18 20:16:15.056733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:15.056395 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:15.073958 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:15.073905 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" podStartSLOduration=0.984591634 podStartE2EDuration="2.073887305s" totalImagesPullingTime="1.089295671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:15.071780339 +0000 UTC m=+894.122140811" watchObservedRunningTime="2026-09-18 20:16:15.073887305 +0000 UTC m=+894.124247751" Sep 18 20:16:19.022720 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.022690 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:19.211210 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.211173 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-587bdb7f54-cgd9v"] Sep 18 20:16:19.213632 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.213607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:19.222737 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.222706 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-587bdb7f54-cgd9v"] Sep 18 20:16:19.271439 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.271397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gwk9\" (UniqueName: \"kubernetes.io/projected/3cbd8d09-10b4-4080-9a86-897781329049-kube-api-access-9gwk9\") pod \"ai-gateway-controller-587bdb7f54-cgd9v\" (UID: \"3cbd8d09-10b4-4080-9a86-897781329049\") " pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:19.372294 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.372241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gwk9\" (UniqueName: \"kubernetes.io/projected/3cbd8d09-10b4-4080-9a86-897781329049-kube-api-access-9gwk9\") pod \"ai-gateway-controller-587bdb7f54-cgd9v\" (UID: \"3cbd8d09-10b4-4080-9a86-897781329049\") " pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:19.380913 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.380887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gwk9\" (UniqueName: \"kubernetes.io/projected/3cbd8d09-10b4-4080-9a86-897781329049-kube-api-access-9gwk9\") pod \"ai-gateway-controller-587bdb7f54-cgd9v\" (UID: \"3cbd8d09-10b4-4080-9a86-897781329049\") " pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:19.525237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.525197 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:19.794867 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:19.794841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-587bdb7f54-cgd9v"] Sep 18 20:16:19.796797 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:19.796769 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbd8d09_10b4_4080_9a86_897781329049.slice/crio-50301f7d3b5a9233685b61ec00f97cab0884da405461960320f98b3513ecadf4 WatchSource:0}: Error finding container 50301f7d3b5a9233685b61ec00f97cab0884da405461960320f98b3513ecadf4: Status 404 returned error can't find the container with id 50301f7d3b5a9233685b61ec00f97cab0884da405461960320f98b3513ecadf4 Sep 18 20:16:20.081641 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:20.081542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" event={"ID":"3cbd8d09-10b4-4080-9a86-897781329049","Type":"ContainerStarted","Data":"50301f7d3b5a9233685b61ec00f97cab0884da405461960320f98b3513ecadf4"} Sep 18 20:16:21.087897 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:21.087860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" event={"ID":"3cbd8d09-10b4-4080-9a86-897781329049","Type":"ContainerStarted","Data":"f48bff79117e3c636a169e99f147f5a991ebf9075ad7bcab2458a4f24c9f532f"} Sep 18 20:16:21.088421 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:21.087959 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:21.104244 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:21.104169 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" podStartSLOduration=1.745736271 podStartE2EDuration="2.104150457s" totalImagesPullingTime="358.414186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:21.102927608 +0000 UTC m=+900.153288057" watchObservedRunningTime="2026-09-18 20:16:21.104150457 +0000 UTC m=+900.154510904" Sep 18 20:16:23.154617 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:23.154585 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:16:23.156419 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:23.156395 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:16:26.064440 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:26.064406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-85fd65c94c-7wlzn" Sep 18 20:16:32.095756 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.095723 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-587bdb7f54-cgd9v" Sep 18 20:16:32.133910 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.133874 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6db44cc5dd-8scfh"] Sep 18 20:16:32.134202 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.134148 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" podUID="857cf02a-c311-4a2d-adb2-b17227f3f5a8" containerName="manager" containerID="cri-o://dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0" gracePeriod=30 Sep 18 20:16:32.389712 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.389686 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:32.490931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.490895 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsvhk\" (UniqueName: \"kubernetes.io/projected/857cf02a-c311-4a2d-adb2-b17227f3f5a8-kube-api-access-bsvhk\") pod \"857cf02a-c311-4a2d-adb2-b17227f3f5a8\" (UID: \"857cf02a-c311-4a2d-adb2-b17227f3f5a8\") " Sep 18 20:16:32.493011 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.492989 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857cf02a-c311-4a2d-adb2-b17227f3f5a8-kube-api-access-bsvhk" (OuterVolumeSpecName: "kube-api-access-bsvhk") pod "857cf02a-c311-4a2d-adb2-b17227f3f5a8" (UID: "857cf02a-c311-4a2d-adb2-b17227f3f5a8"). InnerVolumeSpecName "kube-api-access-bsvhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:16:32.591862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:32.591812 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsvhk\" (UniqueName: \"kubernetes.io/projected/857cf02a-c311-4a2d-adb2-b17227f3f5a8-kube-api-access-bsvhk\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:16:33.156210 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.156170 2559 generic.go:356] "Generic (PLEG): container finished" podID="857cf02a-c311-4a2d-adb2-b17227f3f5a8" containerID="dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0" exitCode=0 Sep 18 20:16:33.156731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.156250 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" Sep 18 20:16:33.156731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.156256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" event={"ID":"857cf02a-c311-4a2d-adb2-b17227f3f5a8","Type":"ContainerDied","Data":"dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0"} Sep 18 20:16:33.156731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.156365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db44cc5dd-8scfh" event={"ID":"857cf02a-c311-4a2d-adb2-b17227f3f5a8","Type":"ContainerDied","Data":"c6910a2122935dde4b733069473106a3650552d9a3e14a0be6d04019310c1d72"} Sep 18 20:16:33.156731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.156389 2559 scope.go:117] "RemoveContainer" containerID="dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0" Sep 18 20:16:33.165268 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.165246 2559 scope.go:117] "RemoveContainer" containerID="dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0" Sep 18 20:16:33.165598 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:16:33.165578 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0\": container with ID starting with dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0 not found: ID does not exist" containerID="dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0" Sep 18 20:16:33.165646 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.165608 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0"} err="failed to get container status \"dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0\": rpc error: code = NotFound desc = could not find container \"dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0\": container with ID starting with dae1de89a5417cd102b8192dc55889af12537d6ad84ee7b3d7d353c50b4d61e0 not found: ID does not exist" Sep 18 20:16:33.176968 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.176936 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6db44cc5dd-8scfh"] Sep 18 20:16:33.183861 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:33.183836 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6db44cc5dd-8scfh"] Sep 18 20:16:35.183541 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:35.183504 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857cf02a-c311-4a2d-adb2-b17227f3f5a8" path="/var/lib/kubelet/pods/857cf02a-c311-4a2d-adb2-b17227f3f5a8/volumes" Sep 18 20:16:55.408580 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.408541 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z"] Sep 18 20:16:55.411528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.409120 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857cf02a-c311-4a2d-adb2-b17227f3f5a8" containerName="manager" Sep 18 20:16:55.411528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.409136 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="857cf02a-c311-4a2d-adb2-b17227f3f5a8" containerName="manager" Sep 18 20:16:55.411528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.409222 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="857cf02a-c311-4a2d-adb2-b17227f3f5a8" containerName="manager" Sep 18 20:16:55.412552 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.412535 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.415178 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.415156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b88002-serving-cert\"" Sep 18 20:16:55.425520 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.425490 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z"] Sep 18 20:16:55.597471 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.597432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.597471 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.597469 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-metrics-certs\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.597718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.597489 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2z2v\" (UniqueName: \"kubernetes.io/projected/204e9330-c547-4463-aad1-45c2daf84d1d-kube-api-access-l2z2v\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.597718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.597588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9330-c547-4463-aad1-45c2daf84d1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.699165 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.699061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.699165 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.699106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-metrics-certs\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.699165 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.699130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2z2v\" (UniqueName: \"kubernetes.io/projected/204e9330-c547-4463-aad1-45c2daf84d1d-kube-api-access-l2z2v\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.699515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.699208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9330-c547-4463-aad1-45c2daf84d1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.699515 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:16:55.699233 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-serving-cert: secret "maas-api-e2e-worker-w3-b88002-serving-cert" not found Sep 18 20:16:55.699515 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:16:55.699360 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls podName:204e9330-c547-4463-aad1-45c2daf84d1d nodeName:}" failed. No retries permitted until 2026-09-18 20:16:56.199336532 +0000 UTC m=+935.249696972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls") pod "maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" (UID: "204e9330-c547-4463-aad1-45c2daf84d1d") : secret "maas-api-e2e-worker-w3-b88002-serving-cert" not found Sep 18 20:16:55.699927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.699909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9330-c547-4463-aad1-45c2daf84d1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.701705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.701685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-metrics-certs\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:55.710008 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:55.709978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2z2v\" (UniqueName: \"kubernetes.io/projected/204e9330-c547-4463-aad1-45c2daf84d1d-kube-api-access-l2z2v\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:56.095246 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.095163 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh"] Sep 18 20:16:56.098766 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.098744 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.101424 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.101404 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b88002\"" Sep 18 20:16:56.101615 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.101595 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b88002-dockercfg-nxvd8\"" Sep 18 20:16:56.112487 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.112457 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh"] Sep 18 20:16:56.122775 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.122743 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls"] Sep 18 20:16:56.126978 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.126950 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.137153 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.137124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls"] Sep 18 20:16:56.203774 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.203735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:56.203774 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.203787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqckb\" (UniqueName: \"kubernetes.io/projected/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-kube-api-access-tqckb\") pod \"payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.204038 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.203904 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.206581 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.206558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:56.305504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.305461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs978\" (UniqueName: \"kubernetes.io/projected/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-kube-api-access-vs978\") pod \"payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.305504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.305519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.305740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.305635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqckb\" (UniqueName: \"kubernetes.io/projected/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-kube-api-access-tqckb\") pod \"payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.305740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.305667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.306412 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.306394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.314749 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.314724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqckb\" (UniqueName: \"kubernetes.io/projected/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-kube-api-access-tqckb\") pod \"payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.334551 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.334510 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:56.406554 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.406518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.406736 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.406614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs978\" (UniqueName: \"kubernetes.io/projected/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-kube-api-access-vs978\") pod \"payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.407499 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.407456 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.409451 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.409429 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:56.415407 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.415379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs978\" (UniqueName: \"kubernetes.io/projected/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-kube-api-access-vs978\") pod \"payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.439307 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.439251 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:56.608926 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.608899 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z"] Sep 18 20:16:56.611436 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:56.611400 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204e9330_c547_4463_aad1_45c2daf84d1d.slice/crio-12fcf3cf8ff51f77ec3d8c27ef8a2ceb43862f9d01fc23fe0ad132b1bd60d2b5 WatchSource:0}: Error finding container 12fcf3cf8ff51f77ec3d8c27ef8a2ceb43862f9d01fc23fe0ad132b1bd60d2b5: Status 404 returned error can't find the container with id 12fcf3cf8ff51f77ec3d8c27ef8a2ceb43862f9d01fc23fe0ad132b1bd60d2b5 Sep 18 20:16:56.693436 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.693349 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh"] Sep 18 20:16:56.696824 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:56.696791 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428d079a_a82c_48c1_b7c3_7cf817a5d5b3.slice/crio-4298817aa7e30880895e8be722b3c1dcdd9e7dac349e94c526790c2e2efaeded WatchSource:0}: Error finding container 4298817aa7e30880895e8be722b3c1dcdd9e7dac349e94c526790c2e2efaeded: Status 404 returned error can't find the container with id 4298817aa7e30880895e8be722b3c1dcdd9e7dac349e94c526790c2e2efaeded Sep 18 20:16:56.728024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:56.727992 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls"] Sep 18 20:16:56.729399 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:56.729368 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce01f2e8_08a9_40dc_bc48_6326a1d536c2.slice/crio-9b68b3e43a0b103cd4f78243c38446634ec0b50c6ca91f9647812f340fe0f8c3 WatchSource:0}: Error finding container 9b68b3e43a0b103cd4f78243c38446634ec0b50c6ca91f9647812f340fe0f8c3: Status 404 returned error can't find the container with id 9b68b3e43a0b103cd4f78243c38446634ec0b50c6ca91f9647812f340fe0f8c3 Sep 18 20:16:57.094483 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.094410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2"] Sep 18 20:16:57.098869 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.098839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.101515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.101487 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e2dea3-serving-cert\"" Sep 18 20:16:57.108988 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.108958 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2"] Sep 18 20:16:57.215664 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.215627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.215856 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.215693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84d01e1-e4c3-49b5-91d5-0981238554a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.215856 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.215762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dzq\" (UniqueName: \"kubernetes.io/projected/b84d01e1-e4c3-49b5-91d5-0981238554a5-kube-api-access-b7dzq\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.215856 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.215808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-metrics-certs\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.280854 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.280747 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" event={"ID":"428d079a-a82c-48c1-b7c3-7cf817a5d5b3","Type":"ContainerStarted","Data":"c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d"} Sep 18 20:16:57.281028 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.280953 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:16:57.281028 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.280995 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" event={"ID":"428d079a-a82c-48c1-b7c3-7cf817a5d5b3","Type":"ContainerStarted","Data":"4298817aa7e30880895e8be722b3c1dcdd9e7dac349e94c526790c2e2efaeded"} Sep 18 20:16:57.282492 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.282469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" event={"ID":"ce01f2e8-08a9-40dc-bc48-6326a1d536c2","Type":"ContainerStarted","Data":"f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805"} Sep 18 20:16:57.282620 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.282519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" event={"ID":"ce01f2e8-08a9-40dc-bc48-6326a1d536c2","Type":"ContainerStarted","Data":"9b68b3e43a0b103cd4f78243c38446634ec0b50c6ca91f9647812f340fe0f8c3"} Sep 18 20:16:57.282620 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.282546 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:16:57.284070 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.284041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" event={"ID":"204e9330-c547-4463-aad1-45c2daf84d1d","Type":"ContainerStarted","Data":"65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59"} Sep 18 20:16:57.284196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.284079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" event={"ID":"204e9330-c547-4463-aad1-45c2daf84d1d","Type":"ContainerStarted","Data":"12fcf3cf8ff51f77ec3d8c27ef8a2ceb43862f9d01fc23fe0ad132b1bd60d2b5"} Sep 18 20:16:57.284196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.284178 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:16:57.300838 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.300787 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" podStartSLOduration=1.300770808 podStartE2EDuration="1.300770808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:57.298674542 +0000 UTC m=+936.349035012" watchObservedRunningTime="2026-09-18 20:16:57.300770808 +0000 UTC m=+936.351131256" Sep 18 20:16:57.315416 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.315362 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" podStartSLOduration=1.946093877 podStartE2EDuration="2.315344101s" totalImagesPullingTime="369.250224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:57.314700737 +0000 UTC m=+936.365061186" watchObservedRunningTime="2026-09-18 20:16:57.315344101 +0000 UTC m=+936.365704548" Sep 18 20:16:57.317146 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.317119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.317233 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.317194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84d01e1-e4c3-49b5-91d5-0981238554a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.317301 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:16:57.317256 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-serving-cert: secret "maas-api-e2e-inf-a-e2dea3-serving-cert" not found Sep 18 20:16:57.317301 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.317292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dzq\" (UniqueName: \"kubernetes.io/projected/b84d01e1-e4c3-49b5-91d5-0981238554a5-kube-api-access-b7dzq\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.317405 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:16:57.317346 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls podName:b84d01e1-e4c3-49b5-91d5-0981238554a5 nodeName:}" failed. No retries permitted until 2026-09-18 20:16:57.817328018 +0000 UTC m=+936.867688448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls") pod "maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" (UID: "b84d01e1-e4c3-49b5-91d5-0981238554a5") : secret "maas-api-e2e-inf-a-e2dea3-serving-cert" not found Sep 18 20:16:57.317405 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.317370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-metrics-certs\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.318046 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.318024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84d01e1-e4c3-49b5-91d5-0981238554a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.319844 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.319826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-metrics-certs\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.330184 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.330148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dzq\" (UniqueName: \"kubernetes.io/projected/b84d01e1-e4c3-49b5-91d5-0981238554a5-kube-api-access-b7dzq\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.335045 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.334988 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" podStartSLOduration=1.334965581 podStartE2EDuration="1.334965581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:57.332720794 +0000 UTC m=+936.383081240" watchObservedRunningTime="2026-09-18 20:16:57.334965581 +0000 UTC m=+936.385326031" Sep 18 20:16:57.822452 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.822411 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:57.825905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:57.825872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:58.012613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.012575 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:58.095347 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.095310 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz"] Sep 18 20:16:58.101196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.101159 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.105148 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.105111 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-116778-serving-cert\"" Sep 18 20:16:58.108454 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.108422 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz"] Sep 18 20:16:58.226259 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.226206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.226503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.226354 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf92k\" (UniqueName: \"kubernetes.io/projected/bfe524e5-7db0-4000-b70a-2e589815d91d-kube-api-access-xf92k\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.226503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.226424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-metrics-certs\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.226503 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.226484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe524e5-7db0-4000-b70a-2e589815d91d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.298479 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.298448 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2"] Sep 18 20:16:58.299854 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:58.299823 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84d01e1_e4c3_49b5_91d5_0981238554a5.slice/crio-269074f977d22675d3180e51701aa632075021cf809f7a10659f76717e6793b3 WatchSource:0}: Error finding container 269074f977d22675d3180e51701aa632075021cf809f7a10659f76717e6793b3: Status 404 returned error can't find the container with id 269074f977d22675d3180e51701aa632075021cf809f7a10659f76717e6793b3 Sep 18 20:16:58.327024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.326921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf92k\" (UniqueName: \"kubernetes.io/projected/bfe524e5-7db0-4000-b70a-2e589815d91d-kube-api-access-xf92k\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.327024 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.326981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-metrics-certs\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.327295 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.327032 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe524e5-7db0-4000-b70a-2e589815d91d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.327295 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.327103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.327841 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.327809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe524e5-7db0-4000-b70a-2e589815d91d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.329664 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.329644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.329769 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.329688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-metrics-certs\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.335074 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.335051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf92k\" (UniqueName: \"kubernetes.io/projected/bfe524e5-7db0-4000-b70a-2e589815d91d-kube-api-access-xf92k\") pod \"maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.417189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.417149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:58.693703 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.693669 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz"] Sep 18 20:16:58.697489 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:58.697436 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe524e5_7db0_4000_b70a_2e589815d91d.slice/crio-8ac1eee13442647093dfb8b31363b4b8201d3124e60074345f511d0fdbb27893 WatchSource:0}: Error finding container 8ac1eee13442647093dfb8b31363b4b8201d3124e60074345f511d0fdbb27893: Status 404 returned error can't find the container with id 8ac1eee13442647093dfb8b31363b4b8201d3124e60074345f511d0fdbb27893 Sep 18 20:16:58.825063 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.825030 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz"] Sep 18 20:16:58.828730 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.828713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:58.831719 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.831695 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-116778-dockercfg-krlnb\"" Sep 18 20:16:58.831873 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.831740 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-116778\"" Sep 18 20:16:58.838831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.838800 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz"] Sep 18 20:16:58.934227 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.934134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98b5a71-2794-46a1-a3be-ac4ade059e3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:58.934227 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:58.934183 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dl9m\" (UniqueName: \"kubernetes.io/projected/a98b5a71-2794-46a1-a3be-ac4ade059e3d-kube-api-access-2dl9m\") pod \"payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:59.035723 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.035668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98b5a71-2794-46a1-a3be-ac4ade059e3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:59.035897 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.035734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dl9m\" (UniqueName: \"kubernetes.io/projected/a98b5a71-2794-46a1-a3be-ac4ade059e3d-kube-api-access-2dl9m\") pod \"payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:59.036853 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.036823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98b5a71-2794-46a1-a3be-ac4ade059e3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:59.043968 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.043940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dl9m\" (UniqueName: \"kubernetes.io/projected/a98b5a71-2794-46a1-a3be-ac4ade059e3d-kube-api-access-2dl9m\") pod \"payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:59.141104 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.141069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:16:59.297026 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.296994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" event={"ID":"b84d01e1-e4c3-49b5-91d5-0981238554a5","Type":"ContainerStarted","Data":"83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7"} Sep 18 20:16:59.297235 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.297047 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" event={"ID":"b84d01e1-e4c3-49b5-91d5-0981238554a5","Type":"ContainerStarted","Data":"269074f977d22675d3180e51701aa632075021cf809f7a10659f76717e6793b3"} Sep 18 20:16:59.297235 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.297100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:16:59.298757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.298728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" event={"ID":"bfe524e5-7db0-4000-b70a-2e589815d91d","Type":"ContainerStarted","Data":"4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1"} Sep 18 20:16:59.298848 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.298784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" event={"ID":"bfe524e5-7db0-4000-b70a-2e589815d91d","Type":"ContainerStarted","Data":"8ac1eee13442647093dfb8b31363b4b8201d3124e60074345f511d0fdbb27893"} Sep 18 20:16:59.298848 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.298814 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:16:59.333812 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.333752 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" podStartSLOduration=1.9825110000000001 podStartE2EDuration="2.333735304s" totalImagesPullingTime="351.224304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:59.332464676 +0000 UTC m=+938.382825124" watchObservedRunningTime="2026-09-18 20:16:59.333735304 +0000 UTC m=+938.384095752" Sep 18 20:16:59.355439 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.355386 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" podStartSLOduration=1.009173822 podStartE2EDuration="1.355365615s" totalImagesPullingTime="346.191793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:59.354497812 +0000 UTC m=+938.404858261" watchObservedRunningTime="2026-09-18 20:16:59.355365615 +0000 UTC m=+938.405726092" Sep 18 20:16:59.430787 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:16:59.430759 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz"] Sep 18 20:16:59.432411 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:16:59.432378 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98b5a71_2794_46a1_a3be_ac4ade059e3d.slice/crio-74ac36f768fe59234ecd5bb05d594ab1b46331b1a0786d25ec30ef77e7725974 WatchSource:0}: Error finding container 74ac36f768fe59234ecd5bb05d594ab1b46331b1a0786d25ec30ef77e7725974: Status 404 returned error can't find the container with id 74ac36f768fe59234ecd5bb05d594ab1b46331b1a0786d25ec30ef77e7725974 Sep 18 20:17:00.306453 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:00.306410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" event={"ID":"a98b5a71-2794-46a1-a3be-ac4ade059e3d","Type":"ContainerStarted","Data":"2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f"} Sep 18 20:17:00.306959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:00.306471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" event={"ID":"a98b5a71-2794-46a1-a3be-ac4ade059e3d","Type":"ContainerStarted","Data":"74ac36f768fe59234ecd5bb05d594ab1b46331b1a0786d25ec30ef77e7725974"} Sep 18 20:17:00.329244 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:00.329167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" podStartSLOduration=2.329143827 podStartE2EDuration="2.329143827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:00.325398418 +0000 UTC m=+939.375758864" watchObservedRunningTime="2026-09-18 20:17:00.329143827 +0000 UTC m=+939.379504277" Sep 18 20:17:01.311802 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:01.311768 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:17:03.295450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:03.295423 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:17:05.312245 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:05.312217 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:17:05.312711 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:05.312670 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:17:07.801169 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.801139 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj"] Sep 18 20:17:07.804723 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.804699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:07.807414 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.807390 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7626fb93-serving-cert\"" Sep 18 20:17:07.814316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.814116 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj"] Sep 18 20:17:07.919209 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.919168 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-metrics-certs\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:07.919424 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.919226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dppc\" (UniqueName: \"kubernetes.io/projected/a6942209-ba0e-4d72-8a8a-80f992425a8f-kube-api-access-9dppc\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:07.919424 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.919379 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6942209-ba0e-4d72-8a8a-80f992425a8f-openshift-service-ca\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:07.919528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:07.919488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-maas-api-tls\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.020414 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.020348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-metrics-certs\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.020414 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.020408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dppc\" (UniqueName: \"kubernetes.io/projected/a6942209-ba0e-4d72-8a8a-80f992425a8f-kube-api-access-9dppc\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.020705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.020490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6942209-ba0e-4d72-8a8a-80f992425a8f-openshift-service-ca\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.020705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.020585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-maas-api-tls\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.021346 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.021319 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6942209-ba0e-4d72-8a8a-80f992425a8f-openshift-service-ca\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.023001 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.022973 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-metrics-certs\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.023136 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.023116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-maas-api-tls\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.028642 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.028614 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dppc\" (UniqueName: \"kubernetes.io/projected/a6942209-ba0e-4d72-8a8a-80f992425a8f-kube-api-access-9dppc\") pod \"maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.118620 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.118585 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:08.291560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.291528 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:17:08.291795 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.291742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:17:08.397650 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:08.397472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj"] Sep 18 20:17:08.400779 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:17:08.400742 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6942209_ba0e_4d72_8a8a_80f992425a8f.slice/crio-36bd5b0d561b74b4aeab4c02a5ca4785a7d0dc8c121902968265502a5a7c987a WatchSource:0}: Error finding container 36bd5b0d561b74b4aeab4c02a5ca4785a7d0dc8c121902968265502a5a7c987a: Status 404 returned error can't find the container with id 36bd5b0d561b74b4aeab4c02a5ca4785a7d0dc8c121902968265502a5a7c987a Sep 18 20:17:09.368146 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:09.368111 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" event={"ID":"a6942209-ba0e-4d72-8a8a-80f992425a8f","Type":"ContainerStarted","Data":"f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626"} Sep 18 20:17:09.368583 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:09.368167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" event={"ID":"a6942209-ba0e-4d72-8a8a-80f992425a8f","Type":"ContainerStarted","Data":"36bd5b0d561b74b4aeab4c02a5ca4785a7d0dc8c121902968265502a5a7c987a"} Sep 18 20:17:09.368583 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:09.368218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:09.400495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:09.400435 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" podStartSLOduration=2.122248456 podStartE2EDuration="2.400415599s" totalImagesPullingTime="278.167143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:09.399360916 +0000 UTC m=+948.449721364" watchObservedRunningTime="2026-09-18 20:17:09.400415599 +0000 UTC m=+948.450776047" Sep 18 20:17:12.320023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:12.319991 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:17:15.379487 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:15.379453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:37.636528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.636309 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj"] Sep 18 20:17:37.637507 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.637445 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" podUID="a6942209-ba0e-4d72-8a8a-80f992425a8f" containerName="maas-api" containerID="cri-o://f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626" gracePeriod=30 Sep 18 20:17:37.903540 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.903514 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:37.928462 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.928422 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-metrics-certs\") pod \"a6942209-ba0e-4d72-8a8a-80f992425a8f\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " Sep 18 20:17:37.928462 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.928460 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dppc\" (UniqueName: \"kubernetes.io/projected/a6942209-ba0e-4d72-8a8a-80f992425a8f-kube-api-access-9dppc\") pod \"a6942209-ba0e-4d72-8a8a-80f992425a8f\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " Sep 18 20:17:37.928757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.928482 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6942209-ba0e-4d72-8a8a-80f992425a8f-openshift-service-ca\") pod \"a6942209-ba0e-4d72-8a8a-80f992425a8f\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " Sep 18 20:17:37.928757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.928506 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-maas-api-tls\") pod \"a6942209-ba0e-4d72-8a8a-80f992425a8f\" (UID: \"a6942209-ba0e-4d72-8a8a-80f992425a8f\") " Sep 18 20:17:37.929439 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.929266 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6942209-ba0e-4d72-8a8a-80f992425a8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6942209-ba0e-4d72-8a8a-80f992425a8f" (UID: "a6942209-ba0e-4d72-8a8a-80f992425a8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:17:37.931253 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.931223 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a6942209-ba0e-4d72-8a8a-80f992425a8f" (UID: "a6942209-ba0e-4d72-8a8a-80f992425a8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:17:37.931383 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.931248 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6942209-ba0e-4d72-8a8a-80f992425a8f" (UID: "a6942209-ba0e-4d72-8a8a-80f992425a8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:17:37.931383 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:37.931255 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6942209-ba0e-4d72-8a8a-80f992425a8f-kube-api-access-9dppc" (OuterVolumeSpecName: "kube-api-access-9dppc") pod "a6942209-ba0e-4d72-8a8a-80f992425a8f" (UID: "a6942209-ba0e-4d72-8a8a-80f992425a8f"). InnerVolumeSpecName "kube-api-access-9dppc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:17:38.030258 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.030220 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:17:38.030485 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.030318 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dppc\" (UniqueName: \"kubernetes.io/projected/a6942209-ba0e-4d72-8a8a-80f992425a8f-kube-api-access-9dppc\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:17:38.030485 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.030340 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6942209-ba0e-4d72-8a8a-80f992425a8f-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:17:38.030485 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.030354 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6942209-ba0e-4d72-8a8a-80f992425a8f-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:17:38.535561 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.535524 2559 generic.go:356] "Generic (PLEG): container finished" podID="a6942209-ba0e-4d72-8a8a-80f992425a8f" containerID="f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626" exitCode=0 Sep 18 20:17:38.535744 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.535572 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" event={"ID":"a6942209-ba0e-4d72-8a8a-80f992425a8f","Type":"ContainerDied","Data":"f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626"} Sep 18 20:17:38.535744 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.535613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" event={"ID":"a6942209-ba0e-4d72-8a8a-80f992425a8f","Type":"ContainerDied","Data":"36bd5b0d561b74b4aeab4c02a5ca4785a7d0dc8c121902968265502a5a7c987a"} Sep 18 20:17:38.535744 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.535633 2559 scope.go:117] "RemoveContainer" containerID="f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626" Sep 18 20:17:38.535744 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.535576 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj" Sep 18 20:17:38.545663 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.545644 2559 scope.go:117] "RemoveContainer" containerID="f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626" Sep 18 20:17:38.545960 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:17:38.545938 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626\": container with ID starting with f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626 not found: ID does not exist" containerID="f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626" Sep 18 20:17:38.546012 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.545970 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626"} err="failed to get container status \"f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626\": rpc error: code = NotFound desc = could not find container \"f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626\": container with ID starting with f5681e5a414d19be082cee067e49f2207efcdeb580196d1ea05f1178addd7626 not found: ID does not exist" Sep 18 20:17:38.557927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.557894 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj"] Sep 18 20:17:38.563073 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:38.563042 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7626fb93-5c8dcdcf89-xvwnj"] Sep 18 20:17:39.185185 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:39.185142 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6942209-ba0e-4d72-8a8a-80f992425a8f" path="/var/lib/kubelet/pods/a6942209-ba0e-4d72-8a8a-80f992425a8f/volumes" Sep 18 20:17:42.480542 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.480504 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v"] Sep 18 20:17:42.481015 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.480915 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6942209-ba0e-4d72-8a8a-80f992425a8f" containerName="maas-api" Sep 18 20:17:42.481015 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.480925 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6942209-ba0e-4d72-8a8a-80f992425a8f" containerName="maas-api" Sep 18 20:17:42.481129 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.481021 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6942209-ba0e-4d72-8a8a-80f992425a8f" containerName="maas-api" Sep 18 20:17:42.485856 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.485821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.490461 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.489513 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6d00b9e3-serving-cert\"" Sep 18 20:17:42.502531 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.502500 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v"] Sep 18 20:17:42.574805 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.574771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbstr\" (UniqueName: \"kubernetes.io/projected/0f151cef-13ef-48d9-8d2a-b7edb9218960-kube-api-access-cbstr\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.574985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.574841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.574985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.574868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-metrics-certs\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.574985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.574917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f151cef-13ef-48d9-8d2a-b7edb9218960-openshift-service-ca\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.675798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.675757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.675798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.675805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-metrics-certs\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.676014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.675869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f151cef-13ef-48d9-8d2a-b7edb9218960-openshift-service-ca\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.676014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.675893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbstr\" (UniqueName: \"kubernetes.io/projected/0f151cef-13ef-48d9-8d2a-b7edb9218960-kube-api-access-cbstr\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.676014 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:17:42.675915 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-serving-cert: secret "maas-api-e2e-migrate-6d00b9e3-serving-cert" not found Sep 18 20:17:42.676014 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:17:42.676011 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls podName:0f151cef-13ef-48d9-8d2a-b7edb9218960 nodeName:}" failed. No retries permitted until 2026-09-18 20:17:43.175987623 +0000 UTC m=+982.226348069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls") pod "maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" (UID: "0f151cef-13ef-48d9-8d2a-b7edb9218960") : secret "maas-api-e2e-migrate-6d00b9e3-serving-cert" not found Sep 18 20:17:42.676614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.676593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f151cef-13ef-48d9-8d2a-b7edb9218960-openshift-service-ca\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.678645 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.678621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-metrics-certs\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:42.687201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:42.687170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbstr\" (UniqueName: \"kubernetes.io/projected/0f151cef-13ef-48d9-8d2a-b7edb9218960-kube-api-access-cbstr\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:43.180606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:43.180563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:43.183116 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:43.183092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls\") pod \"maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:43.398431 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:43.398402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:43.677054 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:43.677030 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v"] Sep 18 20:17:43.679485 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:17:43.679452 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f151cef_13ef_48d9_8d2a_b7edb9218960.slice/crio-c9dd52bdb075dc01e6cdacfaf52c8d3737653ec4133e2773087f8bd218960c5a WatchSource:0}: Error finding container c9dd52bdb075dc01e6cdacfaf52c8d3737653ec4133e2773087f8bd218960c5a: Status 404 returned error can't find the container with id c9dd52bdb075dc01e6cdacfaf52c8d3737653ec4133e2773087f8bd218960c5a Sep 18 20:17:44.572028 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:44.571991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" event={"ID":"0f151cef-13ef-48d9-8d2a-b7edb9218960","Type":"ContainerStarted","Data":"dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa"} Sep 18 20:17:44.572232 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:44.572052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" event={"ID":"0f151cef-13ef-48d9-8d2a-b7edb9218960","Type":"ContainerStarted","Data":"c9dd52bdb075dc01e6cdacfaf52c8d3737653ec4133e2773087f8bd218960c5a"} Sep 18 20:17:44.572232 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:44.572073 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:17:44.591723 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:44.591665 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" podStartSLOduration=2.22385639 podStartE2EDuration="2.59165045s" totalImagesPullingTime="367.79406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:44.590880937 +0000 UTC m=+983.641241385" watchObservedRunningTime="2026-09-18 20:17:44.59165045 +0000 UTC m=+983.642010897" Sep 18 20:17:50.582902 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:17:50.582872 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:18:13.168145 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.168104 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v"] Sep 18 20:18:13.168667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.168406 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" podUID="0f151cef-13ef-48d9-8d2a-b7edb9218960" containerName="maas-api" containerID="cri-o://dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa" gracePeriod=30 Sep 18 20:18:13.435785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.435755 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:18:13.493292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.493244 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-metrics-certs\") pod \"0f151cef-13ef-48d9-8d2a-b7edb9218960\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " Sep 18 20:18:13.493481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.493314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f151cef-13ef-48d9-8d2a-b7edb9218960-openshift-service-ca\") pod \"0f151cef-13ef-48d9-8d2a-b7edb9218960\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " Sep 18 20:18:13.493481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.493374 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbstr\" (UniqueName: \"kubernetes.io/projected/0f151cef-13ef-48d9-8d2a-b7edb9218960-kube-api-access-cbstr\") pod \"0f151cef-13ef-48d9-8d2a-b7edb9218960\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " Sep 18 20:18:13.493481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.493422 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls\") pod \"0f151cef-13ef-48d9-8d2a-b7edb9218960\" (UID: \"0f151cef-13ef-48d9-8d2a-b7edb9218960\") " Sep 18 20:18:13.493725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.493698 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f151cef-13ef-48d9-8d2a-b7edb9218960-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f151cef-13ef-48d9-8d2a-b7edb9218960" (UID: "0f151cef-13ef-48d9-8d2a-b7edb9218960"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:18:13.495927 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.495890 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f151cef-13ef-48d9-8d2a-b7edb9218960" (UID: "0f151cef-13ef-48d9-8d2a-b7edb9218960"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:18:13.496048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.495980 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f151cef-13ef-48d9-8d2a-b7edb9218960" (UID: "0f151cef-13ef-48d9-8d2a-b7edb9218960"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:18:13.496218 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.496195 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f151cef-13ef-48d9-8d2a-b7edb9218960-kube-api-access-cbstr" (OuterVolumeSpecName: "kube-api-access-cbstr") pod "0f151cef-13ef-48d9-8d2a-b7edb9218960" (UID: "0f151cef-13ef-48d9-8d2a-b7edb9218960"). InnerVolumeSpecName "kube-api-access-cbstr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:18:13.594563 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.594527 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:13.594563 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.594556 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f151cef-13ef-48d9-8d2a-b7edb9218960-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:13.594563 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.594565 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbstr\" (UniqueName: \"kubernetes.io/projected/0f151cef-13ef-48d9-8d2a-b7edb9218960-kube-api-access-cbstr\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:13.594563 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.594574 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f151cef-13ef-48d9-8d2a-b7edb9218960-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:13.728072 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.727989 2559 generic.go:356] "Generic (PLEG): container finished" podID="0f151cef-13ef-48d9-8d2a-b7edb9218960" containerID="dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa" exitCode=0 Sep 18 20:18:13.728072 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.728040 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" Sep 18 20:18:13.728315 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.728070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" event={"ID":"0f151cef-13ef-48d9-8d2a-b7edb9218960","Type":"ContainerDied","Data":"dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa"} Sep 18 20:18:13.728315 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.728128 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v" event={"ID":"0f151cef-13ef-48d9-8d2a-b7edb9218960","Type":"ContainerDied","Data":"c9dd52bdb075dc01e6cdacfaf52c8d3737653ec4133e2773087f8bd218960c5a"} Sep 18 20:18:13.728315 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.728151 2559 scope.go:117] "RemoveContainer" containerID="dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa" Sep 18 20:18:13.742107 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.742078 2559 scope.go:117] "RemoveContainer" containerID="dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa" Sep 18 20:18:13.742651 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:18:13.742586 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa\": container with ID starting with dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa not found: ID does not exist" containerID="dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa" Sep 18 20:18:13.742765 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.742652 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa"} err="failed to get container status \"dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa\": rpc error: code = NotFound desc = could not find container \"dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa\": container with ID starting with dfba69e80a49fb1b515384aac857094721deaf083c3ef006b9def8d1013d91fa not found: ID does not exist" Sep 18 20:18:13.755084 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.755055 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v"] Sep 18 20:18:13.764124 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:13.764094 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d00b9e3-757d459986-mrl4v"] Sep 18 20:18:15.184143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:15.184109 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f151cef-13ef-48d9-8d2a-b7edb9218960" path="/var/lib/kubelet/pods/0f151cef-13ef-48d9-8d2a-b7edb9218960/volumes" Sep 18 20:18:21.275694 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.275651 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2"] Sep 18 20:18:21.276112 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.276086 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f151cef-13ef-48d9-8d2a-b7edb9218960" containerName="maas-api" Sep 18 20:18:21.276112 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.276098 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f151cef-13ef-48d9-8d2a-b7edb9218960" containerName="maas-api" Sep 18 20:18:21.276186 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.276180 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f151cef-13ef-48d9-8d2a-b7edb9218960" containerName="maas-api" Sep 18 20:18:21.280698 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.280679 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.283255 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.283231 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f88d7657-dockercfg-q75hh\"" Sep 18 20:18:21.283255 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.283247 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f88d7657\"" Sep 18 20:18:21.290255 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.290229 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2"] Sep 18 20:18:21.295827 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.295802 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7"] Sep 18 20:18:21.299662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.299641 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.310079 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.310049 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7"] Sep 18 20:18:21.369543 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.369508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hq5\" (UniqueName: \"kubernetes.io/projected/a9e5c137-35e1-46c5-afed-cafabd06dff2-kube-api-access-46hq5\") pod \"payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.369731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.369553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.369731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.369703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxgs\" (UniqueName: \"kubernetes.io/projected/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-kube-api-access-zxxgs\") pod \"payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.369832 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.369762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5c137-35e1-46c5-afed-cafabd06dff2-plugins-config-volume\") pod \"payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.470583 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.470535 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.470789 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.470659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxgs\" (UniqueName: \"kubernetes.io/projected/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-kube-api-access-zxxgs\") pod \"payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.470789 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.470702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5c137-35e1-46c5-afed-cafabd06dff2-plugins-config-volume\") pod \"payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.470789 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.470751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hq5\" (UniqueName: \"kubernetes.io/projected/a9e5c137-35e1-46c5-afed-cafabd06dff2-kube-api-access-46hq5\") pod \"payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.471510 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.471486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.471628 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.471607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5c137-35e1-46c5-afed-cafabd06dff2-plugins-config-volume\") pod \"payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.478998 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.478972 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxgs\" (UniqueName: \"kubernetes.io/projected/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-kube-api-access-zxxgs\") pod \"payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.479116 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.479081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hq5\" (UniqueName: \"kubernetes.io/projected/a9e5c137-35e1-46c5-afed-cafabd06dff2-kube-api-access-46hq5\") pod \"payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.592892 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.592803 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:21.612711 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.612683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:21.871819 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.871790 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2"] Sep 18 20:18:21.873964 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:18:21.873937 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8c04df_7ec2_40e2_b9e6_9f9f9b20be21.slice/crio-0cf306633410341a8fd9aff86260a2eb71f6f3b7a30ed39954316836ef5fb580 WatchSource:0}: Error finding container 0cf306633410341a8fd9aff86260a2eb71f6f3b7a30ed39954316836ef5fb580: Status 404 returned error can't find the container with id 0cf306633410341a8fd9aff86260a2eb71f6f3b7a30ed39954316836ef5fb580 Sep 18 20:18:21.898335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:21.898309 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7"] Sep 18 20:18:21.910619 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:18:21.910584 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e5c137_35e1_46c5_afed_cafabd06dff2.slice/crio-e936dd09b793ba020a3901a17c7b1f717821ebcaf743e4200790e27ccf8c1ff0 WatchSource:0}: Error finding container e936dd09b793ba020a3901a17c7b1f717821ebcaf743e4200790e27ccf8c1ff0: Status 404 returned error can't find the container with id e936dd09b793ba020a3901a17c7b1f717821ebcaf743e4200790e27ccf8c1ff0 Sep 18 20:18:22.781481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.781446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" event={"ID":"a9e5c137-35e1-46c5-afed-cafabd06dff2","Type":"ContainerStarted","Data":"09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340"} Sep 18 20:18:22.781916 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.781499 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" event={"ID":"a9e5c137-35e1-46c5-afed-cafabd06dff2","Type":"ContainerStarted","Data":"e936dd09b793ba020a3901a17c7b1f717821ebcaf743e4200790e27ccf8c1ff0"} Sep 18 20:18:22.781916 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.781553 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:22.783215 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.783188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" event={"ID":"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21","Type":"ContainerStarted","Data":"fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff"} Sep 18 20:18:22.783385 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.783233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" event={"ID":"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21","Type":"ContainerStarted","Data":"0cf306633410341a8fd9aff86260a2eb71f6f3b7a30ed39954316836ef5fb580"} Sep 18 20:18:22.783385 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.783256 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:22.800838 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.800726 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" podStartSLOduration=1.800690176 podStartE2EDuration="1.800690176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:22.797730495 +0000 UTC m=+1021.848090945" watchObservedRunningTime="2026-09-18 20:18:22.800690176 +0000 UTC m=+1021.851050637" Sep 18 20:18:22.820131 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:22.820073 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" podStartSLOduration=1.8200553240000001 podStartE2EDuration="1.820055324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:22.819005373 +0000 UTC m=+1021.869365837" watchObservedRunningTime="2026-09-18 20:18:22.820055324 +0000 UTC m=+1021.870415772" Sep 18 20:18:33.790466 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:33.790428 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:33.790858 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:33.790507 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:59.035947 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.035910 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7"] Sep 18 20:18:59.036591 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.036180 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" podUID="a9e5c137-35e1-46c5-afed-cafabd06dff2" containerName="payload-processing" containerID="cri-o://09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340" gracePeriod=30 Sep 18 20:18:59.043332 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.043299 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2"] Sep 18 20:18:59.043566 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.043517 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" podUID="8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" containerName="payload-pre-processing" containerID="cri-o://fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff" gracePeriod=30 Sep 18 20:18:59.303009 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.302984 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:59.335921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.335886 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:59.349040 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.349007 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5c137-35e1-46c5-afed-cafabd06dff2-plugins-config-volume\") pod \"a9e5c137-35e1-46c5-afed-cafabd06dff2\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " Sep 18 20:18:59.349225 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.349112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46hq5\" (UniqueName: \"kubernetes.io/projected/a9e5c137-35e1-46c5-afed-cafabd06dff2-kube-api-access-46hq5\") pod \"a9e5c137-35e1-46c5-afed-cafabd06dff2\" (UID: \"a9e5c137-35e1-46c5-afed-cafabd06dff2\") " Sep 18 20:18:59.349516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.349480 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e5c137-35e1-46c5-afed-cafabd06dff2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9e5c137-35e1-46c5-afed-cafabd06dff2" (UID: "a9e5c137-35e1-46c5-afed-cafabd06dff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:18:59.351235 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.351206 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e5c137-35e1-46c5-afed-cafabd06dff2-kube-api-access-46hq5" (OuterVolumeSpecName: "kube-api-access-46hq5") pod "a9e5c137-35e1-46c5-afed-cafabd06dff2" (UID: "a9e5c137-35e1-46c5-afed-cafabd06dff2"). InnerVolumeSpecName "kube-api-access-46hq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:18:59.450094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.450058 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-plugins-config-volume\") pod \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " Sep 18 20:18:59.450094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.450102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxxgs\" (UniqueName: \"kubernetes.io/projected/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-kube-api-access-zxxgs\") pod \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\" (UID: \"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21\") " Sep 18 20:18:59.450387 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.450315 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5c137-35e1-46c5-afed-cafabd06dff2-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:59.450387 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.450335 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hq5\" (UniqueName: \"kubernetes.io/projected/a9e5c137-35e1-46c5-afed-cafabd06dff2-kube-api-access-46hq5\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:59.450565 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.450536 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" (UID: "8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:18:59.452385 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.452357 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-kube-api-access-zxxgs" (OuterVolumeSpecName: "kube-api-access-zxxgs") pod "8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" (UID: "8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21"). InnerVolumeSpecName "kube-api-access-zxxgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:18:59.551153 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.551064 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:59.551153 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.551092 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxgs\" (UniqueName: \"kubernetes.io/projected/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21-kube-api-access-zxxgs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:18:59.984474 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.984442 2559 generic.go:356] "Generic (PLEG): container finished" podID="a9e5c137-35e1-46c5-afed-cafabd06dff2" containerID="09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340" exitCode=0 Sep 18 20:18:59.984674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.984486 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" Sep 18 20:18:59.984674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.984514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" event={"ID":"a9e5c137-35e1-46c5-afed-cafabd06dff2","Type":"ContainerDied","Data":"09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340"} Sep 18 20:18:59.984674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.984560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7" event={"ID":"a9e5c137-35e1-46c5-afed-cafabd06dff2","Type":"ContainerDied","Data":"e936dd09b793ba020a3901a17c7b1f717821ebcaf743e4200790e27ccf8c1ff0"} Sep 18 20:18:59.984674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.984588 2559 scope.go:117] "RemoveContainer" containerID="09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340" Sep 18 20:18:59.985883 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.985859 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" containerID="fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff" exitCode=0 Sep 18 20:18:59.985968 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.985895 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" Sep 18 20:18:59.985968 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.985948 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" event={"ID":"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21","Type":"ContainerDied","Data":"fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff"} Sep 18 20:18:59.986050 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.986001 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2" event={"ID":"8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21","Type":"ContainerDied","Data":"0cf306633410341a8fd9aff86260a2eb71f6f3b7a30ed39954316836ef5fb580"} Sep 18 20:18:59.995147 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.995129 2559 scope.go:117] "RemoveContainer" containerID="09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340" Sep 18 20:18:59.995568 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:18:59.995532 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340\": container with ID starting with 09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340 not found: ID does not exist" containerID="09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340" Sep 18 20:18:59.995648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.995574 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340"} err="failed to get container status \"09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340\": rpc error: code = NotFound desc = could not find container \"09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340\": container with ID starting with 09906ba7c660d76752df8b4340d902ae16192619984fc43ecd9694ea2113b340 not found: ID does not exist" Sep 18 20:18:59.995648 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:18:59.995593 2559 scope.go:117] "RemoveContainer" containerID="fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff" Sep 18 20:19:00.004739 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:00.004714 2559 scope.go:117] "RemoveContainer" containerID="fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff" Sep 18 20:19:00.005087 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:19:00.005066 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff\": container with ID starting with fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff not found: ID does not exist" containerID="fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff" Sep 18 20:19:00.005178 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:00.005095 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff"} err="failed to get container status \"fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff\": rpc error: code = NotFound desc = could not find container \"fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff\": container with ID starting with fb4a37f29dab16d3410ff48fdf2e8989c100572572dc80113dbea35e6f2406ff not found: ID does not exist" Sep 18 20:19:00.009970 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:00.009935 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7"] Sep 18 20:19:00.012212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:00.012185 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f88d7657-696b88d7df-q2cx7"] Sep 18 20:19:00.022015 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:00.021983 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2"] Sep 18 20:19:00.025954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:00.025923 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f88d7657-764846fc8b-frwt2"] Sep 18 20:19:01.184256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:01.184218 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" path="/var/lib/kubelet/pods/8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21/volumes" Sep 18 20:19:01.184742 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:01.184682 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e5c137-35e1-46c5-afed-cafabd06dff2" path="/var/lib/kubelet/pods/a9e5c137-35e1-46c5-afed-cafabd06dff2/volumes" Sep 18 20:19:07.194152 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194048 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn"] Sep 18 20:19:07.194750 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194727 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e5c137-35e1-46c5-afed-cafabd06dff2" containerName="payload-processing" Sep 18 20:19:07.194796 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194757 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e5c137-35e1-46c5-afed-cafabd06dff2" containerName="payload-processing" Sep 18 20:19:07.194835 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194799 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" containerName="payload-pre-processing" Sep 18 20:19:07.194835 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194809 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" containerName="payload-pre-processing" Sep 18 20:19:07.194922 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194909 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8c04df-7ec2-40e2-b9e6-9f9f9b20be21" containerName="payload-pre-processing" Sep 18 20:19:07.194956 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.194932 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e5c137-35e1-46c5-afed-cafabd06dff2" containerName="payload-processing" Sep 18 20:19:07.200202 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.200171 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.202969 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.202939 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6e56ebc0-serving-cert\"" Sep 18 20:19:07.214327 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.214295 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn"] Sep 18 20:19:07.325761 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.325716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.325947 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.325790 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprfh\" (UniqueName: \"kubernetes.io/projected/56de23f1-4618-44b3-94eb-422c3cfc0a00-kube-api-access-hprfh\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.325947 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.325833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-metrics-certs\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.325947 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.325857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56de23f1-4618-44b3-94eb-422c3cfc0a00-openshift-service-ca\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.426876 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.426832 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.427089 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.426907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprfh\" (UniqueName: \"kubernetes.io/projected/56de23f1-4618-44b3-94eb-422c3cfc0a00-kube-api-access-hprfh\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.427089 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.426948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-metrics-certs\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.427089 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.426981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56de23f1-4618-44b3-94eb-422c3cfc0a00-openshift-service-ca\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.427089 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:19:07.426990 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-serving-cert: secret "maas-api-e2e-derive-6e56ebc0-serving-cert" not found Sep 18 20:19:07.427089 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:19:07.427071 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls podName:56de23f1-4618-44b3-94eb-422c3cfc0a00 nodeName:}" failed. No retries permitted until 2026-09-18 20:19:07.927052598 +0000 UTC m=+1066.977413024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls") pod "maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" (UID: "56de23f1-4618-44b3-94eb-422c3cfc0a00") : secret "maas-api-e2e-derive-6e56ebc0-serving-cert" not found Sep 18 20:19:07.427732 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.427709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56de23f1-4618-44b3-94eb-422c3cfc0a00-openshift-service-ca\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.429548 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.429526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-metrics-certs\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.441527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.441498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprfh\" (UniqueName: \"kubernetes.io/projected/56de23f1-4618-44b3-94eb-422c3cfc0a00-kube-api-access-hprfh\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.931632 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.931591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:07.934225 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:07.934189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls\") pod \"maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:08.122672 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:08.122622 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:08.396694 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:08.396619 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn"] Sep 18 20:19:08.399315 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:19:08.399266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56de23f1_4618_44b3_94eb_422c3cfc0a00.slice/crio-8770b00cf2d369dfec713176f6a074d7d0ae78d70f4cbc4020a7f4f8d2e070cf WatchSource:0}: Error finding container 8770b00cf2d369dfec713176f6a074d7d0ae78d70f4cbc4020a7f4f8d2e070cf: Status 404 returned error can't find the container with id 8770b00cf2d369dfec713176f6a074d7d0ae78d70f4cbc4020a7f4f8d2e070cf Sep 18 20:19:08.400688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:08.400669 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:19:09.040450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:09.040362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" event={"ID":"56de23f1-4618-44b3-94eb-422c3cfc0a00","Type":"ContainerStarted","Data":"f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a"} Sep 18 20:19:09.040450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:09.040415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" event={"ID":"56de23f1-4618-44b3-94eb-422c3cfc0a00","Type":"ContainerStarted","Data":"8770b00cf2d369dfec713176f6a074d7d0ae78d70f4cbc4020a7f4f8d2e070cf"} Sep 18 20:19:09.040635 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:09.040478 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:09.059288 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:09.059214 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" podStartSLOduration=1.7207029 podStartE2EDuration="2.059197854s" totalImagesPullingTime="338.494954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:09.057318332 +0000 UTC m=+1068.107678777" watchObservedRunningTime="2026-09-18 20:19:09.059197854 +0000 UTC m=+1068.109558302" Sep 18 20:19:15.054830 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:15.054791 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:45.750870 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:45.750830 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn"] Sep 18 20:19:45.751541 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:45.751324 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" podUID="56de23f1-4618-44b3-94eb-422c3cfc0a00" containerName="maas-api" containerID="cri-o://f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a" gracePeriod=30 Sep 18 20:19:46.017094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.017025 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:46.081561 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.081528 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls\") pod \"56de23f1-4618-44b3-94eb-422c3cfc0a00\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " Sep 18 20:19:46.081785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.081572 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hprfh\" (UniqueName: \"kubernetes.io/projected/56de23f1-4618-44b3-94eb-422c3cfc0a00-kube-api-access-hprfh\") pod \"56de23f1-4618-44b3-94eb-422c3cfc0a00\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " Sep 18 20:19:46.081785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.081593 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56de23f1-4618-44b3-94eb-422c3cfc0a00-openshift-service-ca\") pod \"56de23f1-4618-44b3-94eb-422c3cfc0a00\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " Sep 18 20:19:46.081785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.081622 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-metrics-certs\") pod \"56de23f1-4618-44b3-94eb-422c3cfc0a00\" (UID: \"56de23f1-4618-44b3-94eb-422c3cfc0a00\") " Sep 18 20:19:46.082014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.081967 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56de23f1-4618-44b3-94eb-422c3cfc0a00-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56de23f1-4618-44b3-94eb-422c3cfc0a00" (UID: "56de23f1-4618-44b3-94eb-422c3cfc0a00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:19:46.083866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.083832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56de23f1-4618-44b3-94eb-422c3cfc0a00-kube-api-access-hprfh" (OuterVolumeSpecName: "kube-api-access-hprfh") pod "56de23f1-4618-44b3-94eb-422c3cfc0a00" (UID: "56de23f1-4618-44b3-94eb-422c3cfc0a00"). InnerVolumeSpecName "kube-api-access-hprfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:19:46.083997 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.083939 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56de23f1-4618-44b3-94eb-422c3cfc0a00" (UID: "56de23f1-4618-44b3-94eb-422c3cfc0a00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:19:46.083997 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.083953 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56de23f1-4618-44b3-94eb-422c3cfc0a00" (UID: "56de23f1-4618-44b3-94eb-422c3cfc0a00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:19:46.183065 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.183031 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:19:46.183065 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.183056 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56de23f1-4618-44b3-94eb-422c3cfc0a00-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:19:46.183065 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.183066 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hprfh\" (UniqueName: \"kubernetes.io/projected/56de23f1-4618-44b3-94eb-422c3cfc0a00-kube-api-access-hprfh\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:19:46.183065 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.183076 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56de23f1-4618-44b3-94eb-422c3cfc0a00-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:19:46.241258 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.241228 2559 generic.go:356] "Generic (PLEG): container finished" podID="56de23f1-4618-44b3-94eb-422c3cfc0a00" containerID="f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a" exitCode=0 Sep 18 20:19:46.241438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.241296 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" Sep 18 20:19:46.241438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.241296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" event={"ID":"56de23f1-4618-44b3-94eb-422c3cfc0a00","Type":"ContainerDied","Data":"f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a"} Sep 18 20:19:46.241438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.241417 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn" event={"ID":"56de23f1-4618-44b3-94eb-422c3cfc0a00","Type":"ContainerDied","Data":"8770b00cf2d369dfec713176f6a074d7d0ae78d70f4cbc4020a7f4f8d2e070cf"} Sep 18 20:19:46.241438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.241439 2559 scope.go:117] "RemoveContainer" containerID="f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a" Sep 18 20:19:46.251831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.251807 2559 scope.go:117] "RemoveContainer" containerID="f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a" Sep 18 20:19:46.252373 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:19:46.252344 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a\": container with ID starting with f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a not found: ID does not exist" containerID="f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a" Sep 18 20:19:46.252373 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.252386 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a"} err="failed to get container status \"f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a\": rpc error: code = NotFound desc = could not find container \"f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a\": container with ID starting with f75541158eb83b611e57d1359aa0afb830d5a23fdfe609a52516ccf270139f1a not found: ID does not exist" Sep 18 20:19:46.267879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.267795 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn"] Sep 18 20:19:46.271042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:46.271019 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e56ebc0-5f6f4c5675-lg5vn"] Sep 18 20:19:47.184122 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:19:47.184082 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56de23f1-4618-44b3-94eb-422c3cfc0a00" path="/var/lib/kubelet/pods/56de23f1-4618-44b3-94eb-422c3cfc0a00/volumes" Sep 18 20:20:28.092206 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.092158 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn"] Sep 18 20:20:28.092842 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.092820 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56de23f1-4618-44b3-94eb-422c3cfc0a00" containerName="maas-api" Sep 18 20:20:28.092928 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.092845 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="56de23f1-4618-44b3-94eb-422c3cfc0a00" containerName="maas-api" Sep 18 20:20:28.093042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.092948 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="56de23f1-4618-44b3-94eb-422c3cfc0a00" containerName="maas-api" Sep 18 20:20:28.097699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.097665 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.101619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.101588 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b88002\"/\"distinct-w3-b88002-kserve-self-signed-certs\"" Sep 18 20:20:28.101619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.101611 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b88002\"/\"default-dockercfg-hdqwp\"" Sep 18 20:20:28.101855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.101605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b88002\"/\"kube-root-ca.crt\"" Sep 18 20:20:28.101855 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.101591 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b88002\"/\"openshift-service-ca.crt\"" Sep 18 20:20:28.106483 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.106453 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn"] Sep 18 20:20:28.173449 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.173415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpql\" (UniqueName: \"kubernetes.io/projected/eca5a948-2e49-4f36-8916-6502191284a6-kube-api-access-xnpql\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.173637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.173499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-dshm\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.173637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.173522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca5a948-2e49-4f36-8916-6502191284a6-tls-certs\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.173637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.173603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-model-cache\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.173778 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.173640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-tmp-dir\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.173778 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.173666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-home\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275011 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpql\" (UniqueName: \"kubernetes.io/projected/eca5a948-2e49-4f36-8916-6502191284a6-kube-api-access-xnpql\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-dshm\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca5a948-2e49-4f36-8916-6502191284a6-tls-certs\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275140 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-model-cache\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-tmp-dir\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275189 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-home\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-model-cache\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275727 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-tmp-dir\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.275727 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.275686 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-home\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.277491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.277470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-dshm\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.277874 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.277853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca5a948-2e49-4f36-8916-6502191284a6-tls-certs\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.293724 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.293687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpql\" (UniqueName: \"kubernetes.io/projected/eca5a948-2e49-4f36-8916-6502191284a6-kube-api-access-xnpql\") pod \"distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.409765 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.409711 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:28.686238 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:28.686094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn"] Sep 18 20:20:28.688600 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:20:28.688570 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeca5a948_2e49_4f36_8916_6502191284a6.slice/crio-b67bc93b448dcda7c507709614c4a09757e5182bdf9f8725c7270c05c370c27f WatchSource:0}: Error finding container b67bc93b448dcda7c507709614c4a09757e5182bdf9f8725c7270c05c370c27f: Status 404 returned error can't find the container with id b67bc93b448dcda7c507709614c4a09757e5182bdf9f8725c7270c05c370c27f Sep 18 20:20:29.479641 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:29.479607 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" event={"ID":"eca5a948-2e49-4f36-8916-6502191284a6","Type":"ContainerStarted","Data":"50710cbd22744c102c9f3221ac55be74cca5782e32a4aac7553acd644593bcd3"} Sep 18 20:20:29.479641 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:29.479649 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" event={"ID":"eca5a948-2e49-4f36-8916-6502191284a6","Type":"ContainerStarted","Data":"b67bc93b448dcda7c507709614c4a09757e5182bdf9f8725c7270c05c370c27f"} Sep 18 20:20:29.496725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:29.496666 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" podStartSLOduration=1.496647783 podStartE2EDuration="1.496647783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:29.494895672 +0000 UTC m=+1148.545256120" watchObservedRunningTime="2026-09-18 20:20:29.496647783 +0000 UTC m=+1148.547008231" Sep 18 20:20:38.410270 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:38.410158 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:38.410270 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:38.410249 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:38.423208 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:38.423177 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:20:38.537888 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:20:38.537856 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:21:05.759483 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.759442 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl"] Sep 18 20:21:05.763647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.763624 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.767688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.767657 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b88002\"/\"distinct-2-w3-b88002-kserve-self-signed-certs\"" Sep 18 20:21:05.779687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.779656 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl"] Sep 18 20:21:05.837463 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.837420 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h69l\" (UniqueName: \"kubernetes.io/projected/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-kube-api-access-7h69l\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.837670 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.837474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-model-cache\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.837670 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.837556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tmp-dir\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.837670 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.837607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-dshm\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.837860 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.837693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-home\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.837860 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.837750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tls-certs\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938473 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938432 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-dshm\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938485 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-home\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tls-certs\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h69l\" (UniqueName: \"kubernetes.io/projected/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-kube-api-access-7h69l\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938576 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-model-cache\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tmp-dir\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.938981 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.938948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-home\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.939090 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.939067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tmp-dir\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.939138 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.939067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-model-cache\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.940853 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.940817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-dshm\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.941177 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.941158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tls-certs\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:05.946668 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:05.946641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h69l\" (UniqueName: \"kubernetes.io/projected/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-kube-api-access-7h69l\") pod \"distinct-2-w3-b88002-kserve-79fb6c87f-cggrl\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:06.075793 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:06.075700 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:06.363572 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:06.363541 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl"] Sep 18 20:21:06.366028 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:21:06.365999 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc6974c5_ec13_4d09_8c40_bbe6ed42b6d3.slice/crio-92f3195649b16ffa8d1d026cb5d0573d6ae5dc1a1e6a7f90812a1df6cab1825c WatchSource:0}: Error finding container 92f3195649b16ffa8d1d026cb5d0573d6ae5dc1a1e6a7f90812a1df6cab1825c: Status 404 returned error can't find the container with id 92f3195649b16ffa8d1d026cb5d0573d6ae5dc1a1e6a7f90812a1df6cab1825c Sep 18 20:21:06.672237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:06.672136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" event={"ID":"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3","Type":"ContainerStarted","Data":"b5e7629037eec9b34531a3096f80193deaf6bd5d068c8db179e9d2236d7668a5"} Sep 18 20:21:06.672237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:06.672185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" event={"ID":"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3","Type":"ContainerStarted","Data":"92f3195649b16ffa8d1d026cb5d0573d6ae5dc1a1e6a7f90812a1df6cab1825c"} Sep 18 20:21:06.689673 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:06.689618 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" podStartSLOduration=1.689602458 podStartE2EDuration="1.689602458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:06.688351169 +0000 UTC m=+1185.738711617" watchObservedRunningTime="2026-09-18 20:21:06.689602458 +0000 UTC m=+1185.739962932" Sep 18 20:21:16.076504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:16.076460 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:16.076504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:16.076518 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:16.089911 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:16.089881 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:16.735577 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:16.735544 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:21:23.202000 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:23.201968 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:21:23.203398 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:23.203328 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:21:27.492597 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.492556 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7"] Sep 18 20:21:27.496682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.496656 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.499400 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.499380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-802c6c2a-serving-cert\"" Sep 18 20:21:27.508003 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.507979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7"] Sep 18 20:21:27.643657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.643620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-metrics-certs\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.643877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.643673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ea265d-089b-4492-aa15-f0e345b8076b-openshift-service-ca\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.643877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.643789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lprxb\" (UniqueName: \"kubernetes.io/projected/09ea265d-089b-4492-aa15-f0e345b8076b-kube-api-access-lprxb\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.643877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.643872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-maas-api-tls\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.744959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.744858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-metrics-certs\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.744959 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.744908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ea265d-089b-4492-aa15-f0e345b8076b-openshift-service-ca\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.745171 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.744987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lprxb\" (UniqueName: \"kubernetes.io/projected/09ea265d-089b-4492-aa15-f0e345b8076b-kube-api-access-lprxb\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.745171 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.745063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-maas-api-tls\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.745748 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.745725 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ea265d-089b-4492-aa15-f0e345b8076b-openshift-service-ca\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.747574 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.747553 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-maas-api-tls\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.747574 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.747564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-metrics-certs\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.756687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.756656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lprxb\" (UniqueName: \"kubernetes.io/projected/09ea265d-089b-4492-aa15-f0e345b8076b-kube-api-access-lprxb\") pod \"maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:27.808350 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:27.808312 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:28.093357 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.093320 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7"] Sep 18 20:21:28.096801 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:21:28.096471 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ea265d_089b_4492_aa15_f0e345b8076b.slice/crio-e03d98a06195a14a61310f825b6a1bacd3427d1cdbdded7bc367d6ffc3a55dc6 WatchSource:0}: Error finding container e03d98a06195a14a61310f825b6a1bacd3427d1cdbdded7bc367d6ffc3a55dc6: Status 404 returned error can't find the container with id e03d98a06195a14a61310f825b6a1bacd3427d1cdbdded7bc367d6ffc3a55dc6 Sep 18 20:21:28.105359 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.105313 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r"] Sep 18 20:21:28.111007 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.110908 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.113631 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.113607 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-802c6c2a-dockercfg-h2lkm\"" Sep 18 20:21:28.113764 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.113680 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-802c6c2a\"" Sep 18 20:21:28.117862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.117838 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r"] Sep 18 20:21:28.250718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.250680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-plugins-config-volume\") pod \"payload-processing-e2e-mt-802c6c2a-774646596f-6745r\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.250920 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.250799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjt9k\" (UniqueName: \"kubernetes.io/projected/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-kube-api-access-rjt9k\") pod \"payload-processing-e2e-mt-802c6c2a-774646596f-6745r\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.351450 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.351422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-plugins-config-volume\") pod \"payload-processing-e2e-mt-802c6c2a-774646596f-6745r\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.351650 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.351524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjt9k\" (UniqueName: \"kubernetes.io/projected/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-kube-api-access-rjt9k\") pod \"payload-processing-e2e-mt-802c6c2a-774646596f-6745r\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.352212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.352191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-plugins-config-volume\") pod \"payload-processing-e2e-mt-802c6c2a-774646596f-6745r\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.361511 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.361490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjt9k\" (UniqueName: \"kubernetes.io/projected/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-kube-api-access-rjt9k\") pod \"payload-processing-e2e-mt-802c6c2a-774646596f-6745r\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.424735 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.424705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:28.713469 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.713445 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r"] Sep 18 20:21:28.716019 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:21:28.715983 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f23caaf_7a65_447b_aa41_8ba5cd8b155d.slice/crio-71ae2c6e0c183eaff671db6b753d60dbca86e278d7235e38e70d570a89688a1a WatchSource:0}: Error finding container 71ae2c6e0c183eaff671db6b753d60dbca86e278d7235e38e70d570a89688a1a: Status 404 returned error can't find the container with id 71ae2c6e0c183eaff671db6b753d60dbca86e278d7235e38e70d570a89688a1a Sep 18 20:21:28.793051 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.793021 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" event={"ID":"5f23caaf-7a65-447b-aa41-8ba5cd8b155d","Type":"ContainerStarted","Data":"71ae2c6e0c183eaff671db6b753d60dbca86e278d7235e38e70d570a89688a1a"} Sep 18 20:21:28.794659 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.794632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" event={"ID":"09ea265d-089b-4492-aa15-f0e345b8076b","Type":"ContainerStarted","Data":"fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038"} Sep 18 20:21:28.794807 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.794684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" event={"ID":"09ea265d-089b-4492-aa15-f0e345b8076b","Type":"ContainerStarted","Data":"e03d98a06195a14a61310f825b6a1bacd3427d1cdbdded7bc367d6ffc3a55dc6"} Sep 18 20:21:28.794807 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.794743 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:28.811532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:28.811470 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" podStartSLOduration=1.46332618 podStartE2EDuration="1.811447758s" totalImagesPullingTime="348.121578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:28.81107606 +0000 UTC m=+1207.861436509" watchObservedRunningTime="2026-09-18 20:21:28.811447758 +0000 UTC m=+1207.861808209" Sep 18 20:21:29.801884 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:29.801849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" event={"ID":"5f23caaf-7a65-447b-aa41-8ba5cd8b155d","Type":"ContainerStarted","Data":"b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303"} Sep 18 20:21:29.819627 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:29.819575 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" podStartSLOduration=1.819559386 podStartE2EDuration="1.819559386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:29.816989687 +0000 UTC m=+1208.867350139" watchObservedRunningTime="2026-09-18 20:21:29.819559386 +0000 UTC m=+1208.869919835" Sep 18 20:21:30.807862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:30.807829 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:21:34.807812 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:34.807784 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:21:41.815767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:21:41.815730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:22:07.559249 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:07.559147 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7"] Sep 18 20:22:07.559904 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:07.559441 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" podUID="09ea265d-089b-4492-aa15-f0e345b8076b" containerName="maas-api" containerID="cri-o://fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038" gracePeriod=30 Sep 18 20:22:07.611969 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:07.611926 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r"] Sep 18 20:22:07.612378 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:07.612211 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" podUID="5f23caaf-7a65-447b-aa41-8ba5cd8b155d" containerName="payload-processing" containerID="cri-o://b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303" gracePeriod=30 Sep 18 20:22:07.888318 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:07.888262 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:22:07.914368 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:07.914341 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:22:08.014463 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.014424 2559 generic.go:356] "Generic (PLEG): container finished" podID="5f23caaf-7a65-447b-aa41-8ba5cd8b155d" containerID="b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303" exitCode=0 Sep 18 20:22:08.014463 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.014473 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" Sep 18 20:22:08.014723 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.014488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" event={"ID":"5f23caaf-7a65-447b-aa41-8ba5cd8b155d","Type":"ContainerDied","Data":"b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303"} Sep 18 20:22:08.014723 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.014635 2559 scope.go:117] "RemoveContainer" containerID="b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303" Sep 18 20:22:08.014816 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.014552 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r" event={"ID":"5f23caaf-7a65-447b-aa41-8ba5cd8b155d","Type":"ContainerDied","Data":"71ae2c6e0c183eaff671db6b753d60dbca86e278d7235e38e70d570a89688a1a"} Sep 18 20:22:08.017062 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.017035 2559 generic.go:356] "Generic (PLEG): container finished" podID="09ea265d-089b-4492-aa15-f0e345b8076b" containerID="fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038" exitCode=0 Sep 18 20:22:08.017193 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.017104 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" Sep 18 20:22:08.017193 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.017106 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" event={"ID":"09ea265d-089b-4492-aa15-f0e345b8076b","Type":"ContainerDied","Data":"fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038"} Sep 18 20:22:08.017307 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.017209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7" event={"ID":"09ea265d-089b-4492-aa15-f0e345b8076b","Type":"ContainerDied","Data":"e03d98a06195a14a61310f825b6a1bacd3427d1cdbdded7bc367d6ffc3a55dc6"} Sep 18 20:22:08.024859 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.024845 2559 scope.go:117] "RemoveContainer" containerID="b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303" Sep 18 20:22:08.025119 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:22:08.025100 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303\": container with ID starting with b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303 not found: ID does not exist" containerID="b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303" Sep 18 20:22:08.025188 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.025127 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303"} err="failed to get container status \"b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303\": rpc error: code = NotFound desc = could not find container \"b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303\": container with ID starting with b61815c1dec93e90f5d2cf1922815a883c92634b3308c8036807013cc0bbb303 not found: ID does not exist" Sep 18 20:22:08.025188 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.025142 2559 scope.go:117] "RemoveContainer" containerID="fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038" Sep 18 20:22:08.033606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.033577 2559 scope.go:117] "RemoveContainer" containerID="fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038" Sep 18 20:22:08.033931 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:22:08.033912 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038\": container with ID starting with fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038 not found: ID does not exist" containerID="fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038" Sep 18 20:22:08.034004 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.033939 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038"} err="failed to get container status \"fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038\": rpc error: code = NotFound desc = could not find container \"fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038\": container with ID starting with fbfb3e452dd56c9f268d312bc5be5bfdb2b92576d188ed573753f59c66093038 not found: ID does not exist" Sep 18 20:22:08.037340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037317 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-metrics-certs\") pod \"09ea265d-089b-4492-aa15-f0e345b8076b\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " Sep 18 20:22:08.037427 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037375 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-maas-api-tls\") pod \"09ea265d-089b-4492-aa15-f0e345b8076b\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " Sep 18 20:22:08.037427 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037401 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-plugins-config-volume\") pod \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " Sep 18 20:22:08.037526 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037432 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjt9k\" (UniqueName: \"kubernetes.io/projected/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-kube-api-access-rjt9k\") pod \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\" (UID: \"5f23caaf-7a65-447b-aa41-8ba5cd8b155d\") " Sep 18 20:22:08.037526 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037468 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lprxb\" (UniqueName: \"kubernetes.io/projected/09ea265d-089b-4492-aa15-f0e345b8076b-kube-api-access-lprxb\") pod \"09ea265d-089b-4492-aa15-f0e345b8076b\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " Sep 18 20:22:08.037640 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037524 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ea265d-089b-4492-aa15-f0e345b8076b-openshift-service-ca\") pod \"09ea265d-089b-4492-aa15-f0e345b8076b\" (UID: \"09ea265d-089b-4492-aa15-f0e345b8076b\") " Sep 18 20:22:08.037833 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.037807 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f23caaf-7a65-447b-aa41-8ba5cd8b155d" (UID: "5f23caaf-7a65-447b-aa41-8ba5cd8b155d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:22:08.038142 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.038111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ea265d-089b-4492-aa15-f0e345b8076b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09ea265d-089b-4492-aa15-f0e345b8076b" (UID: "09ea265d-089b-4492-aa15-f0e345b8076b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:22:08.039849 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.039822 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ea265d-089b-4492-aa15-f0e345b8076b-kube-api-access-lprxb" (OuterVolumeSpecName: "kube-api-access-lprxb") pod "09ea265d-089b-4492-aa15-f0e345b8076b" (UID: "09ea265d-089b-4492-aa15-f0e345b8076b"). InnerVolumeSpecName "kube-api-access-lprxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:22:08.039925 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.039844 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "09ea265d-089b-4492-aa15-f0e345b8076b" (UID: "09ea265d-089b-4492-aa15-f0e345b8076b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:22:08.040075 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.040057 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-kube-api-access-rjt9k" (OuterVolumeSpecName: "kube-api-access-rjt9k") pod "5f23caaf-7a65-447b-aa41-8ba5cd8b155d" (UID: "5f23caaf-7a65-447b-aa41-8ba5cd8b155d"). InnerVolumeSpecName "kube-api-access-rjt9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:22:08.040150 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.040136 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09ea265d-089b-4492-aa15-f0e345b8076b" (UID: "09ea265d-089b-4492-aa15-f0e345b8076b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:22:08.138831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.138796 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ea265d-089b-4492-aa15-f0e345b8076b-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:08.138831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.138828 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:08.138831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.138838 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ea265d-089b-4492-aa15-f0e345b8076b-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:08.139073 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.138846 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:08.139073 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.138855 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjt9k\" (UniqueName: \"kubernetes.io/projected/5f23caaf-7a65-447b-aa41-8ba5cd8b155d-kube-api-access-rjt9k\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:08.139073 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.138863 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lprxb\" (UniqueName: \"kubernetes.io/projected/09ea265d-089b-4492-aa15-f0e345b8076b-kube-api-access-lprxb\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:08.339553 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.339518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r"] Sep 18 20:22:08.343618 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.343588 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-802c6c2a-774646596f-6745r"] Sep 18 20:22:08.353789 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.353739 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7"] Sep 18 20:22:08.361953 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:08.361921 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-802c6c2a-78577f747b-nlsz7"] Sep 18 20:22:09.184718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:09.184682 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ea265d-089b-4492-aa15-f0e345b8076b" path="/var/lib/kubelet/pods/09ea265d-089b-4492-aa15-f0e345b8076b/volumes" Sep 18 20:22:09.185290 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:09.185253 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f23caaf-7a65-447b-aa41-8ba5cd8b155d" path="/var/lib/kubelet/pods/5f23caaf-7a65-447b-aa41-8ba5cd8b155d/volumes" Sep 18 20:22:13.126872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.126834 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2"] Sep 18 20:22:13.127689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.127649 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" podUID="b84d01e1-e4c3-49b5-91d5-0981238554a5" containerName="maas-api" containerID="cri-o://83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7" gracePeriod=30 Sep 18 20:22:13.390628 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.390564 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:22:13.491125 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.491086 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls\") pod \"b84d01e1-e4c3-49b5-91d5-0981238554a5\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " Sep 18 20:22:13.491340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.491201 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7dzq\" (UniqueName: \"kubernetes.io/projected/b84d01e1-e4c3-49b5-91d5-0981238554a5-kube-api-access-b7dzq\") pod \"b84d01e1-e4c3-49b5-91d5-0981238554a5\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " Sep 18 20:22:13.491340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.491247 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84d01e1-e4c3-49b5-91d5-0981238554a5-openshift-service-ca\") pod \"b84d01e1-e4c3-49b5-91d5-0981238554a5\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " Sep 18 20:22:13.491340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.491267 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-metrics-certs\") pod \"b84d01e1-e4c3-49b5-91d5-0981238554a5\" (UID: \"b84d01e1-e4c3-49b5-91d5-0981238554a5\") " Sep 18 20:22:13.491654 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.491627 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84d01e1-e4c3-49b5-91d5-0981238554a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b84d01e1-e4c3-49b5-91d5-0981238554a5" (UID: "b84d01e1-e4c3-49b5-91d5-0981238554a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:22:13.493678 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.493625 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84d01e1-e4c3-49b5-91d5-0981238554a5-kube-api-access-b7dzq" (OuterVolumeSpecName: "kube-api-access-b7dzq") pod "b84d01e1-e4c3-49b5-91d5-0981238554a5" (UID: "b84d01e1-e4c3-49b5-91d5-0981238554a5"). InnerVolumeSpecName "kube-api-access-b7dzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:22:13.493817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.493716 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b84d01e1-e4c3-49b5-91d5-0981238554a5" (UID: "b84d01e1-e4c3-49b5-91d5-0981238554a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:22:13.493817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.493782 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b84d01e1-e4c3-49b5-91d5-0981238554a5" (UID: "b84d01e1-e4c3-49b5-91d5-0981238554a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:22:13.592667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.592631 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:13.592667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.592664 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7dzq\" (UniqueName: \"kubernetes.io/projected/b84d01e1-e4c3-49b5-91d5-0981238554a5-kube-api-access-b7dzq\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:13.592667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.592676 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84d01e1-e4c3-49b5-91d5-0981238554a5-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:13.592667 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:13.592685 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b84d01e1-e4c3-49b5-91d5-0981238554a5-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:22:14.053900 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.053859 2559 generic.go:356] "Generic (PLEG): container finished" podID="b84d01e1-e4c3-49b5-91d5-0981238554a5" containerID="83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7" exitCode=0 Sep 18 20:22:14.054091 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.053912 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" Sep 18 20:22:14.054091 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.053941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" event={"ID":"b84d01e1-e4c3-49b5-91d5-0981238554a5","Type":"ContainerDied","Data":"83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7"} Sep 18 20:22:14.054091 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.053997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2" event={"ID":"b84d01e1-e4c3-49b5-91d5-0981238554a5","Type":"ContainerDied","Data":"269074f977d22675d3180e51701aa632075021cf809f7a10659f76717e6793b3"} Sep 18 20:22:14.054091 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.054020 2559 scope.go:117] "RemoveContainer" containerID="83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7" Sep 18 20:22:14.064263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.064244 2559 scope.go:117] "RemoveContainer" containerID="83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7" Sep 18 20:22:14.064539 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:22:14.064519 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7\": container with ID starting with 83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7 not found: ID does not exist" containerID="83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7" Sep 18 20:22:14.064593 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.064549 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7"} err="failed to get container status \"83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7\": rpc error: code = NotFound desc = could not find container \"83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7\": container with ID starting with 83142b1734f7e0073bca771c8b57d61c2d05ad35ffe9c8d9e2125d6e02785ab7 not found: ID does not exist" Sep 18 20:22:14.080152 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.080119 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2"] Sep 18 20:22:14.084857 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:14.084827 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2dea3-5df8d67d6f-rwzx2"] Sep 18 20:22:15.185500 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:15.185470 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84d01e1-e4c3-49b5-91d5-0981238554a5" path="/var/lib/kubelet/pods/b84d01e1-e4c3-49b5-91d5-0981238554a5/volumes" Sep 18 20:22:33.491832 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.491795 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h"] Sep 18 20:22:33.492488 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492466 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ea265d-089b-4492-aa15-f0e345b8076b" containerName="maas-api" Sep 18 20:22:33.492612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492491 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ea265d-089b-4492-aa15-f0e345b8076b" containerName="maas-api" Sep 18 20:22:33.492612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492539 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84d01e1-e4c3-49b5-91d5-0981238554a5" containerName="maas-api" Sep 18 20:22:33.492612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492549 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84d01e1-e4c3-49b5-91d5-0981238554a5" containerName="maas-api" Sep 18 20:22:33.492612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492564 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f23caaf-7a65-447b-aa41-8ba5cd8b155d" containerName="payload-processing" Sep 18 20:22:33.492612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492574 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f23caaf-7a65-447b-aa41-8ba5cd8b155d" containerName="payload-processing" Sep 18 20:22:33.492831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492665 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f23caaf-7a65-447b-aa41-8ba5cd8b155d" containerName="payload-processing" Sep 18 20:22:33.492831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492684 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ea265d-089b-4492-aa15-f0e345b8076b" containerName="maas-api" Sep 18 20:22:33.492831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.492695 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84d01e1-e4c3-49b5-91d5-0981238554a5" containerName="maas-api" Sep 18 20:22:33.498406 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.498363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.501057 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.501035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-35f893-serving-cert\"" Sep 18 20:22:33.505421 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.505382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h"] Sep 18 20:22:33.571309 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.571243 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faca8468-ffba-4da3-857a-50ee50360016-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.571495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.571333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.571495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.571363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.571495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.571436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnffk\" (UniqueName: \"kubernetes.io/projected/faca8468-ffba-4da3-857a-50ee50360016-kube-api-access-tnffk\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.672676 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.672637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnffk\" (UniqueName: \"kubernetes.io/projected/faca8468-ffba-4da3-857a-50ee50360016-kube-api-access-tnffk\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.672871 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.672803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faca8468-ffba-4da3-857a-50ee50360016-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.672871 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.672859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.672964 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.672889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.673056 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:22:33.673034 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-serving-cert: secret "maas-api-e2e-shared-a-w1-35f893-serving-cert" not found Sep 18 20:22:33.673155 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:22:33.673142 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls podName:faca8468-ffba-4da3-857a-50ee50360016 nodeName:}" failed. No retries permitted until 2026-09-18 20:22:34.173126444 +0000 UTC m=+1273.223486875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls") pod "maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" (UID: "faca8468-ffba-4da3-857a-50ee50360016") : secret "maas-api-e2e-shared-a-w1-35f893-serving-cert" not found Sep 18 20:22:33.673519 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.673498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faca8468-ffba-4da3-857a-50ee50360016-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.675331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.675314 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:33.684068 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:33.684047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnffk\" (UniqueName: \"kubernetes.io/projected/faca8468-ffba-4da3-857a-50ee50360016-kube-api-access-tnffk\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:34.177462 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:34.177422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:34.179918 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:34.179887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:34.413348 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:34.413302 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:34.687485 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:34.687457 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h"] Sep 18 20:22:34.689335 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:22:34.689309 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaca8468_ffba_4da3_857a_50ee50360016.slice/crio-c1409d7656cb3d47c802da21e38f639dc25e5e250f8bf6d5e3c702d45bd8cbcc WatchSource:0}: Error finding container c1409d7656cb3d47c802da21e38f639dc25e5e250f8bf6d5e3c702d45bd8cbcc: Status 404 returned error can't find the container with id c1409d7656cb3d47c802da21e38f639dc25e5e250f8bf6d5e3c702d45bd8cbcc Sep 18 20:22:35.164729 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:35.164695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" event={"ID":"faca8468-ffba-4da3-857a-50ee50360016","Type":"ContainerStarted","Data":"c1409d7656cb3d47c802da21e38f639dc25e5e250f8bf6d5e3c702d45bd8cbcc"} Sep 18 20:22:36.171745 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:36.171699 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" event={"ID":"faca8468-ffba-4da3-857a-50ee50360016","Type":"ContainerStarted","Data":"d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e"} Sep 18 20:22:36.172196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:36.171973 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:36.188834 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:36.188790 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" podStartSLOduration=2.7776610550000003 podStartE2EDuration="3.188774923s" totalImagesPullingTime="411.113868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:36.187567384 +0000 UTC m=+1275.237927834" watchObservedRunningTime="2026-09-18 20:22:36.188774923 +0000 UTC m=+1275.239135370" Sep 18 20:22:42.182714 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:42.182676 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:22:46.949710 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:46.949671 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn"] Sep 18 20:22:46.953651 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:46.953615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:46.956462 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:46.956440 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bab50d\"" Sep 18 20:22:46.966267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:46.966236 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn"] Sep 18 20:22:47.104575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.104533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85df\" (UniqueName: \"kubernetes.io/projected/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-kube-api-access-f85df\") pod \"payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.104898 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.104878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.206129 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.206031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85df\" (UniqueName: \"kubernetes.io/projected/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-kube-api-access-f85df\") pod \"payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.206129 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.206101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.206839 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.206813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.216237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.216202 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85df\" (UniqueName: \"kubernetes.io/projected/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-kube-api-access-f85df\") pod \"payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.264905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.264859 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:47.543460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:47.543432 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn"] Sep 18 20:22:47.546290 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:22:47.546242 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacf64e7_5435_4590_a8c1_7f67bd61d9cc.slice/crio-0622e892db4e0217cea4d2206e79640a206941b94e51ae069bf43c3343529c78 WatchSource:0}: Error finding container 0622e892db4e0217cea4d2206e79640a206941b94e51ae069bf43c3343529c78: Status 404 returned error can't find the container with id 0622e892db4e0217cea4d2206e79640a206941b94e51ae069bf43c3343529c78 Sep 18 20:22:48.237816 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:48.237775 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" event={"ID":"aacf64e7-5435-4590-a8c1-7f67bd61d9cc","Type":"ContainerStarted","Data":"465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060"} Sep 18 20:22:48.238263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:48.237851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" event={"ID":"aacf64e7-5435-4590-a8c1-7f67bd61d9cc","Type":"ContainerStarted","Data":"0622e892db4e0217cea4d2206e79640a206941b94e51ae069bf43c3343529c78"} Sep 18 20:22:48.238263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:48.237983 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:22:48.256903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:48.256840 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" podStartSLOduration=2.256822521 podStartE2EDuration="2.256822521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:48.254382058 +0000 UTC m=+1287.304742506" watchObservedRunningTime="2026-09-18 20:22:48.256822521 +0000 UTC m=+1287.307182969" Sep 18 20:22:51.461830 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.461792 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg"] Sep 18 20:22:51.465786 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.465766 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.468326 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.468300 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9a5190\"" Sep 18 20:22:51.475468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.475441 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg"] Sep 18 20:22:51.489708 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.489678 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5"] Sep 18 20:22:51.493255 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.493236 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.496383 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.496335 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9a5190-dockercfg-29pds\"" Sep 18 20:22:51.500584 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.500557 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5"] Sep 18 20:22:51.549149 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.549105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpxb\" (UniqueName: \"kubernetes.io/projected/f4162009-fa3e-4bba-8239-8d25e81e4471-kube-api-access-ccpxb\") pod \"payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.549378 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.549172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4162009-fa3e-4bba-8239-8d25e81e4471-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.650643 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.650607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjbh\" (UniqueName: \"kubernetes.io/projected/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-kube-api-access-4pjbh\") pod \"payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.650865 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.650662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpxb\" (UniqueName: \"kubernetes.io/projected/f4162009-fa3e-4bba-8239-8d25e81e4471-kube-api-access-ccpxb\") pod \"payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.650865 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.650759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4162009-fa3e-4bba-8239-8d25e81e4471-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.650865 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.650835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.651468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.651448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4162009-fa3e-4bba-8239-8d25e81e4471-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.659044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.659006 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpxb\" (UniqueName: \"kubernetes.io/projected/f4162009-fa3e-4bba-8239-8d25e81e4471-kube-api-access-ccpxb\") pod \"payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.751714 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.751621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.751714 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.751697 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjbh\" (UniqueName: \"kubernetes.io/projected/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-kube-api-access-4pjbh\") pod \"payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.752411 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.752389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.760050 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.760015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjbh\" (UniqueName: \"kubernetes.io/projected/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-kube-api-access-4pjbh\") pod \"payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:51.778527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.778496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:51.806026 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:51.805991 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:52.060156 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.060122 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg"] Sep 18 20:22:52.062090 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:22:52.062063 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4162009_fa3e_4bba_8239_8d25e81e4471.slice/crio-5dbd30ec27ed2b006d4c36522a31cb07a26e7d827324d00ce1f408b6e9fceae6 WatchSource:0}: Error finding container 5dbd30ec27ed2b006d4c36522a31cb07a26e7d827324d00ce1f408b6e9fceae6: Status 404 returned error can't find the container with id 5dbd30ec27ed2b006d4c36522a31cb07a26e7d827324d00ce1f408b6e9fceae6 Sep 18 20:22:52.091396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.091367 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5"] Sep 18 20:22:52.096474 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:22:52.096446 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecb8a7f_2edf_4ebe_b70e_36efd74c399c.slice/crio-de5ecb9bc0a4f1b067866378bee2d8fb794672aa7722b636439083a33ba97d8c WatchSource:0}: Error finding container de5ecb9bc0a4f1b067866378bee2d8fb794672aa7722b636439083a33ba97d8c: Status 404 returned error can't find the container with id de5ecb9bc0a4f1b067866378bee2d8fb794672aa7722b636439083a33ba97d8c Sep 18 20:22:52.262160 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.262118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" event={"ID":"f4162009-fa3e-4bba-8239-8d25e81e4471","Type":"ContainerStarted","Data":"632f2129b2a6e0644d6a1203094f7b5f93e0b37340dc4473b98b93989229f455"} Sep 18 20:22:52.262365 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.262190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:22:52.262365 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.262216 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" event={"ID":"f4162009-fa3e-4bba-8239-8d25e81e4471","Type":"ContainerStarted","Data":"5dbd30ec27ed2b006d4c36522a31cb07a26e7d827324d00ce1f408b6e9fceae6"} Sep 18 20:22:52.268095 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.268062 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" event={"ID":"eecb8a7f-2edf-4ebe-b70e-36efd74c399c","Type":"ContainerStarted","Data":"86225ffbd6eccca9dab69eb2fd00d62808b6325fd642984d759f7c76f49b4ad0"} Sep 18 20:22:52.268288 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.268154 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" event={"ID":"eecb8a7f-2edf-4ebe-b70e-36efd74c399c","Type":"ContainerStarted","Data":"de5ecb9bc0a4f1b067866378bee2d8fb794672aa7722b636439083a33ba97d8c"} Sep 18 20:22:52.268288 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.268189 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:22:52.281970 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.281915 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" podStartSLOduration=1.281896746 podStartE2EDuration="1.281896746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:52.279582064 +0000 UTC m=+1291.329942512" watchObservedRunningTime="2026-09-18 20:22:52.281896746 +0000 UTC m=+1291.332257193" Sep 18 20:22:52.295920 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:52.295865 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" podStartSLOduration=1.2958478709999999 podStartE2EDuration="1.295847871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:52.294858854 +0000 UTC m=+1291.345219303" watchObservedRunningTime="2026-09-18 20:22:52.295847871 +0000 UTC m=+1291.346208356" Sep 18 20:22:59.245793 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:22:59.245755 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:23:03.276027 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:03.275995 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:23:03.276412 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:03.276066 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:23:14.145760 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.145714 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz"] Sep 18 20:23:14.149834 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.149806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.153113 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.153077 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-35558b-dockercfg-db87d\"" Sep 18 20:23:14.153256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.153178 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-35558b\"" Sep 18 20:23:14.160634 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.160602 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz"] Sep 18 20:23:14.267487 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.267424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t997n\" (UniqueName: \"kubernetes.io/projected/bfe34356-d124-4917-b63a-87eb926021aa-kube-api-access-t997n\") pod \"payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.267725 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.267591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe34356-d124-4917-b63a-87eb926021aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.368685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.368643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t997n\" (UniqueName: \"kubernetes.io/projected/bfe34356-d124-4917-b63a-87eb926021aa-kube-api-access-t997n\") pod \"payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.368868 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.368740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe34356-d124-4917-b63a-87eb926021aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.369389 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.369372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe34356-d124-4917-b63a-87eb926021aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.378066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.378036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t997n\" (UniqueName: \"kubernetes.io/projected/bfe34356-d124-4917-b63a-87eb926021aa-kube-api-access-t997n\") pod \"payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.462676 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.462588 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:14.743049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:14.743018 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz"] Sep 18 20:23:14.744604 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:23:14.744573 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe34356_d124_4917_b63a_87eb926021aa.slice/crio-b6f1f1ff66bc3ae7ba9a1809b398efbbadf1de55a054973f2b8d71920eb9fbf8 WatchSource:0}: Error finding container b6f1f1ff66bc3ae7ba9a1809b398efbbadf1de55a054973f2b8d71920eb9fbf8: Status 404 returned error can't find the container with id b6f1f1ff66bc3ae7ba9a1809b398efbbadf1de55a054973f2b8d71920eb9fbf8 Sep 18 20:23:15.404932 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:15.404892 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" event={"ID":"bfe34356-d124-4917-b63a-87eb926021aa","Type":"ContainerStarted","Data":"6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5"} Sep 18 20:23:15.405374 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:15.404957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:15.405374 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:15.404984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" event={"ID":"bfe34356-d124-4917-b63a-87eb926021aa","Type":"ContainerStarted","Data":"b6f1f1ff66bc3ae7ba9a1809b398efbbadf1de55a054973f2b8d71920eb9fbf8"} Sep 18 20:23:15.421415 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:15.421359 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" podStartSLOduration=1.4213419 podStartE2EDuration="1.4213419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:15.420194254 +0000 UTC m=+1314.470554716" watchObservedRunningTime="2026-09-18 20:23:15.4213419 +0000 UTC m=+1314.471702350" Sep 18 20:23:26.420708 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:26.420677 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:23:32.502502 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.502468 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976"] Sep 18 20:23:32.507579 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.507558 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.510532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.510509 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-10f4b77d\"" Sep 18 20:23:32.510674 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.510515 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-10f4b77d-dockercfg-tmz4t\"" Sep 18 20:23:32.515333 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.515304 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976"] Sep 18 20:23:32.646016 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.645965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd24m\" (UniqueName: \"kubernetes.io/projected/6c3ae076-606d-4fa4-b28f-55145d588cef-kube-api-access-vd24m\") pod \"payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.646228 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.646133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ae076-606d-4fa4-b28f-55145d588cef-plugins-config-volume\") pod \"payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.747068 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.747030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ae076-606d-4fa4-b28f-55145d588cef-plugins-config-volume\") pod \"payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.747226 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.747133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd24m\" (UniqueName: \"kubernetes.io/projected/6c3ae076-606d-4fa4-b28f-55145d588cef-kube-api-access-vd24m\") pod \"payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.747806 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.747786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ae076-606d-4fa4-b28f-55145d588cef-plugins-config-volume\") pod \"payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.755714 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.755643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd24m\" (UniqueName: \"kubernetes.io/projected/6c3ae076-606d-4fa4-b28f-55145d588cef-kube-api-access-vd24m\") pod \"payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:32.819819 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:32.819780 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:33.099342 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:33.099210 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976"] Sep 18 20:23:33.102322 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:23:33.102254 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3ae076_606d_4fa4_b28f_55145d588cef.slice/crio-55579992a322f1ac18ecfb22d71b7a2f95adbc26ae1713c3079cc5758c722fb9 WatchSource:0}: Error finding container 55579992a322f1ac18ecfb22d71b7a2f95adbc26ae1713c3079cc5758c722fb9: Status 404 returned error can't find the container with id 55579992a322f1ac18ecfb22d71b7a2f95adbc26ae1713c3079cc5758c722fb9 Sep 18 20:23:33.510916 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:33.510876 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" event={"ID":"6c3ae076-606d-4fa4-b28f-55145d588cef","Type":"ContainerStarted","Data":"43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905"} Sep 18 20:23:33.511419 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:33.510949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" event={"ID":"6c3ae076-606d-4fa4-b28f-55145d588cef","Type":"ContainerStarted","Data":"55579992a322f1ac18ecfb22d71b7a2f95adbc26ae1713c3079cc5758c722fb9"} Sep 18 20:23:33.511419 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:33.511085 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:23:33.528193 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:33.528129 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" podStartSLOduration=1.528111429 podStartE2EDuration="1.528111429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:33.52681597 +0000 UTC m=+1332.577176419" watchObservedRunningTime="2026-09-18 20:23:33.528111429 +0000 UTC m=+1332.578471876" Sep 18 20:23:44.520078 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:23:44.519999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:24:04.590540 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:04.590503 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl"] Sep 18 20:24:04.595104 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:04.595074 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn"] Sep 18 20:24:15.165494 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.165458 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z"] Sep 18 20:24:15.166138 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.165747 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" podUID="204e9330-c547-4463-aad1-45c2daf84d1d" containerName="maas-api" containerID="cri-o://65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59" gracePeriod=30 Sep 18 20:24:15.219999 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.219964 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls"] Sep 18 20:24:15.220232 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.220203 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" podUID="ce01f2e8-08a9-40dc-bc48-6326a1d536c2" containerName="payload-processing" containerID="cri-o://f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805" gracePeriod=30 Sep 18 20:24:15.228102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.228044 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh"] Sep 18 20:24:15.228895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.228430 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" podUID="428d079a-a82c-48c1-b7c3-7cf817a5d5b3" containerName="payload-pre-processing" containerID="cri-o://c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d" gracePeriod=30 Sep 18 20:24:15.553087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.553063 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:24:15.557087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.557058 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:24:15.560953 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.560930 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:24:15.682221 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682179 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9330-c547-4463-aad1-45c2daf84d1d-openshift-service-ca\") pod \"204e9330-c547-4463-aad1-45c2daf84d1d\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " Sep 18 20:24:15.682460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682237 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-metrics-certs\") pod \"204e9330-c547-4463-aad1-45c2daf84d1d\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " Sep 18 20:24:15.682460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682264 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls\") pod \"204e9330-c547-4463-aad1-45c2daf84d1d\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " Sep 18 20:24:15.682460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682294 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2z2v\" (UniqueName: \"kubernetes.io/projected/204e9330-c547-4463-aad1-45c2daf84d1d-kube-api-access-l2z2v\") pod \"204e9330-c547-4463-aad1-45c2daf84d1d\" (UID: \"204e9330-c547-4463-aad1-45c2daf84d1d\") " Sep 18 20:24:15.682460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682327 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-plugins-config-volume\") pod \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " Sep 18 20:24:15.682460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682401 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqckb\" (UniqueName: \"kubernetes.io/projected/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-kube-api-access-tqckb\") pod \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " Sep 18 20:24:15.682460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682426 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-plugins-config-volume\") pod \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\" (UID: \"428d079a-a82c-48c1-b7c3-7cf817a5d5b3\") " Sep 18 20:24:15.682798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682464 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs978\" (UniqueName: \"kubernetes.io/projected/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-kube-api-access-vs978\") pod \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\" (UID: \"ce01f2e8-08a9-40dc-bc48-6326a1d536c2\") " Sep 18 20:24:15.682798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682693 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204e9330-c547-4463-aad1-45c2daf84d1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "204e9330-c547-4463-aad1-45c2daf84d1d" (UID: "204e9330-c547-4463-aad1-45c2daf84d1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:24:15.682798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682725 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce01f2e8-08a9-40dc-bc48-6326a1d536c2" (UID: "ce01f2e8-08a9-40dc-bc48-6326a1d536c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:24:15.682944 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.682841 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204e9330-c547-4463-aad1-45c2daf84d1d-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.683037 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.683013 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "428d079a-a82c-48c1-b7c3-7cf817a5d5b3" (UID: "428d079a-a82c-48c1-b7c3-7cf817a5d5b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:24:15.684691 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.684656 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "204e9330-c547-4463-aad1-45c2daf84d1d" (UID: "204e9330-c547-4463-aad1-45c2daf84d1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:24:15.684794 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.684714 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-kube-api-access-tqckb" (OuterVolumeSpecName: "kube-api-access-tqckb") pod "428d079a-a82c-48c1-b7c3-7cf817a5d5b3" (UID: "428d079a-a82c-48c1-b7c3-7cf817a5d5b3"). InnerVolumeSpecName "kube-api-access-tqckb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:24:15.684794 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.684740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e9330-c547-4463-aad1-45c2daf84d1d-kube-api-access-l2z2v" (OuterVolumeSpecName: "kube-api-access-l2z2v") pod "204e9330-c547-4463-aad1-45c2daf84d1d" (UID: "204e9330-c547-4463-aad1-45c2daf84d1d"). InnerVolumeSpecName "kube-api-access-l2z2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:24:15.684921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.684905 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "204e9330-c547-4463-aad1-45c2daf84d1d" (UID: "204e9330-c547-4463-aad1-45c2daf84d1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:24:15.684995 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.684956 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-kube-api-access-vs978" (OuterVolumeSpecName: "kube-api-access-vs978") pod "ce01f2e8-08a9-40dc-bc48-6326a1d536c2" (UID: "ce01f2e8-08a9-40dc-bc48-6326a1d536c2"). InnerVolumeSpecName "kube-api-access-vs978". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:24:15.749761 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.749647 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce01f2e8-08a9-40dc-bc48-6326a1d536c2" containerID="f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805" exitCode=0 Sep 18 20:24:15.749761 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.749702 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" Sep 18 20:24:15.749761 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.749733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" event={"ID":"ce01f2e8-08a9-40dc-bc48-6326a1d536c2","Type":"ContainerDied","Data":"f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805"} Sep 18 20:24:15.750062 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.749797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls" event={"ID":"ce01f2e8-08a9-40dc-bc48-6326a1d536c2","Type":"ContainerDied","Data":"9b68b3e43a0b103cd4f78243c38446634ec0b50c6ca91f9647812f340fe0f8c3"} Sep 18 20:24:15.750062 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.749825 2559 scope.go:117] "RemoveContainer" containerID="f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805" Sep 18 20:24:15.751368 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.751339 2559 generic.go:356] "Generic (PLEG): container finished" podID="204e9330-c547-4463-aad1-45c2daf84d1d" containerID="65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59" exitCode=0 Sep 18 20:24:15.751497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.751388 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" Sep 18 20:24:15.751497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.751421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" event={"ID":"204e9330-c547-4463-aad1-45c2daf84d1d","Type":"ContainerDied","Data":"65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59"} Sep 18 20:24:15.751497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.751467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z" event={"ID":"204e9330-c547-4463-aad1-45c2daf84d1d","Type":"ContainerDied","Data":"12fcf3cf8ff51f77ec3d8c27ef8a2ceb43862f9d01fc23fe0ad132b1bd60d2b5"} Sep 18 20:24:15.753410 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.753388 2559 generic.go:356] "Generic (PLEG): container finished" podID="428d079a-a82c-48c1-b7c3-7cf817a5d5b3" containerID="c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d" exitCode=0 Sep 18 20:24:15.753532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.753498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" event={"ID":"428d079a-a82c-48c1-b7c3-7cf817a5d5b3","Type":"ContainerDied","Data":"c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d"} Sep 18 20:24:15.753597 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.753537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" event={"ID":"428d079a-a82c-48c1-b7c3-7cf817a5d5b3","Type":"ContainerDied","Data":"4298817aa7e30880895e8be722b3c1dcdd9e7dac349e94c526790c2e2efaeded"} Sep 18 20:24:15.753638 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.753620 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh" Sep 18 20:24:15.759903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.759885 2559 scope.go:117] "RemoveContainer" containerID="f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805" Sep 18 20:24:15.760292 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:24:15.760249 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805\": container with ID starting with f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805 not found: ID does not exist" containerID="f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805" Sep 18 20:24:15.760395 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.760305 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805"} err="failed to get container status \"f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805\": rpc error: code = NotFound desc = could not find container \"f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805\": container with ID starting with f702f42bfa08257c9f964e2722888cf5539a0366182aa986640e58bf2a146805 not found: ID does not exist" Sep 18 20:24:15.760395 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.760324 2559 scope.go:117] "RemoveContainer" containerID="65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59" Sep 18 20:24:15.770575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.770557 2559 scope.go:117] "RemoveContainer" containerID="65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59" Sep 18 20:24:15.770878 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:24:15.770858 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59\": container with ID starting with 65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59 not found: ID does not exist" containerID="65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59" Sep 18 20:24:15.770931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.770886 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59"} err="failed to get container status \"65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59\": rpc error: code = NotFound desc = could not find container \"65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59\": container with ID starting with 65587b7865db383114f253d62617ae4f58684dcb8808672ec463b4fbf3608f59 not found: ID does not exist" Sep 18 20:24:15.770931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.770904 2559 scope.go:117] "RemoveContainer" containerID="c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d" Sep 18 20:24:15.776326 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.776268 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls"] Sep 18 20:24:15.778902 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.778879 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b88002-6fc7c8585c-s59ls"] Sep 18 20:24:15.780863 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.780843 2559 scope.go:117] "RemoveContainer" containerID="c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d" Sep 18 20:24:15.781187 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:24:15.781162 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d\": container with ID starting with c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d not found: ID does not exist" containerID="c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d" Sep 18 20:24:15.781267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.781198 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d"} err="failed to get container status \"c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d\": rpc error: code = NotFound desc = could not find container \"c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d\": container with ID starting with c96d21a06250c89c28a812ed2e7542736b6f9b41776a92f1cfd7afd953cb112d not found: ID does not exist" Sep 18 20:24:15.783975 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.783956 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqckb\" (UniqueName: \"kubernetes.io/projected/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-kube-api-access-tqckb\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.784036 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.783977 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d079a-a82c-48c1-b7c3-7cf817a5d5b3-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.784036 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.783988 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs978\" (UniqueName: \"kubernetes.io/projected/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-kube-api-access-vs978\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.784036 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.783997 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.784036 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.784005 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204e9330-c547-4463-aad1-45c2daf84d1d-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.784036 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.784014 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2z2v\" (UniqueName: \"kubernetes.io/projected/204e9330-c547-4463-aad1-45c2daf84d1d-kube-api-access-l2z2v\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.784036 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.784022 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01f2e8-08a9-40dc-bc48-6326a1d536c2-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:15.792594 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.792558 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z"] Sep 18 20:24:15.802134 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.802105 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b88002-6666765d79-mxm6z"] Sep 18 20:24:15.812244 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.812211 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh"] Sep 18 20:24:15.815926 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:15.815898 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b88002-747cf6f4bb-nzqgh"] Sep 18 20:24:17.184516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:17.184475 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="204e9330-c547-4463-aad1-45c2daf84d1d" path="/var/lib/kubelet/pods/204e9330-c547-4463-aad1-45c2daf84d1d/volumes" Sep 18 20:24:17.185083 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:17.185061 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428d079a-a82c-48c1-b7c3-7cf817a5d5b3" path="/var/lib/kubelet/pods/428d079a-a82c-48c1-b7c3-7cf817a5d5b3/volumes" Sep 18 20:24:17.185507 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:17.185487 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce01f2e8-08a9-40dc-bc48-6326a1d536c2" path="/var/lib/kubelet/pods/ce01f2e8-08a9-40dc-bc48-6326a1d536c2/volumes" Sep 18 20:24:19.611733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.611687 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" podUID="cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" containerName="main" containerID="cri-o://b5e7629037eec9b34531a3096f80193deaf6bd5d068c8db179e9d2236d7668a5" gracePeriod=45 Sep 18 20:24:19.615575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.615538 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" podUID="eca5a948-2e49-4f36-8916-6502191284a6" containerName="main" containerID="cri-o://50710cbd22744c102c9f3221ac55be74cca5782e32a4aac7553acd644593bcd3" gracePeriod=45 Sep 18 20:24:19.790572 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.790538 2559 generic.go:356] "Generic (PLEG): container finished" podID="eca5a948-2e49-4f36-8916-6502191284a6" containerID="50710cbd22744c102c9f3221ac55be74cca5782e32a4aac7553acd644593bcd3" exitCode=0 Sep 18 20:24:19.790733 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.790574 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" event={"ID":"eca5a948-2e49-4f36-8916-6502191284a6","Type":"ContainerDied","Data":"50710cbd22744c102c9f3221ac55be74cca5782e32a4aac7553acd644593bcd3"} Sep 18 20:24:19.792642 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.792614 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" containerID="b5e7629037eec9b34531a3096f80193deaf6bd5d068c8db179e9d2236d7668a5" exitCode=0 Sep 18 20:24:19.792770 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.792659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" event={"ID":"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3","Type":"ContainerDied","Data":"b5e7629037eec9b34531a3096f80193deaf6bd5d068c8db179e9d2236d7668a5"} Sep 18 20:24:19.898651 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.898621 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:24:19.902043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:19.902019 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:24:20.026007 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.025950 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-model-cache\") pod \"eca5a948-2e49-4f36-8916-6502191284a6\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " Sep 18 20:24:20.026007 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026023 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-dshm\") pod \"eca5a948-2e49-4f36-8916-6502191284a6\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026040 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-dshm\") pod \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026059 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-tmp-dir\") pod \"eca5a948-2e49-4f36-8916-6502191284a6\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026120 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-home\") pod \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026155 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnpql\" (UniqueName: \"kubernetes.io/projected/eca5a948-2e49-4f36-8916-6502191284a6-kube-api-access-xnpql\") pod \"eca5a948-2e49-4f36-8916-6502191284a6\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026187 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tmp-dir\") pod \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-model-cache\") pod \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " Sep 18 20:24:20.026313 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026313 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-home\") pod \"eca5a948-2e49-4f36-8916-6502191284a6\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " Sep 18 20:24:20.026669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026336 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "eca5a948-2e49-4f36-8916-6502191284a6" (UID: "eca5a948-2e49-4f36-8916-6502191284a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.026669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026347 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca5a948-2e49-4f36-8916-6502191284a6-tls-certs\") pod \"eca5a948-2e49-4f36-8916-6502191284a6\" (UID: \"eca5a948-2e49-4f36-8916-6502191284a6\") " Sep 18 20:24:20.026669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026383 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eca5a948-2e49-4f36-8916-6502191284a6" (UID: "eca5a948-2e49-4f36-8916-6502191284a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.026669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026423 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h69l\" (UniqueName: \"kubernetes.io/projected/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-kube-api-access-7h69l\") pod \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " Sep 18 20:24:20.026669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026469 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tls-certs\") pod \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\" (UID: \"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3\") " Sep 18 20:24:20.026669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026618 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" (UID: "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.026993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026860 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-model-cache\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.026993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026878 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-tmp-dir\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.026993 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.026895 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-model-cache\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.027149 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.027102 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" (UID: "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.027554 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.027257 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-home" (OuterVolumeSpecName: "home") pod "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" (UID: "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.027554 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.027508 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-home" (OuterVolumeSpecName: "home") pod "eca5a948-2e49-4f36-8916-6502191284a6" (UID: "eca5a948-2e49-4f36-8916-6502191284a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.029007 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.028970 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-dshm" (OuterVolumeSpecName: "dshm") pod "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" (UID: "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.029211 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.029186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca5a948-2e49-4f36-8916-6502191284a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eca5a948-2e49-4f36-8916-6502191284a6" (UID: "eca5a948-2e49-4f36-8916-6502191284a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:24:20.029211 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.029197 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" (UID: "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:24:20.029385 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.029210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-dshm" (OuterVolumeSpecName: "dshm") pod "eca5a948-2e49-4f36-8916-6502191284a6" (UID: "eca5a948-2e49-4f36-8916-6502191284a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:24:20.029385 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.029221 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-kube-api-access-7h69l" (OuterVolumeSpecName: "kube-api-access-7h69l") pod "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" (UID: "cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3"). InnerVolumeSpecName "kube-api-access-7h69l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:24:20.030413 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.030386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca5a948-2e49-4f36-8916-6502191284a6-kube-api-access-xnpql" (OuterVolumeSpecName: "kube-api-access-xnpql") pod "eca5a948-2e49-4f36-8916-6502191284a6" (UID: "eca5a948-2e49-4f36-8916-6502191284a6"). InnerVolumeSpecName "kube-api-access-xnpql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128524 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-home\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128556 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpql\" (UniqueName: \"kubernetes.io/projected/eca5a948-2e49-4f36-8916-6502191284a6-kube-api-access-xnpql\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128568 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tmp-dir\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128577 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-home\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128585 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca5a948-2e49-4f36-8916-6502191284a6-tls-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128592 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h69l\" (UniqueName: \"kubernetes.io/projected/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-kube-api-access-7h69l\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128601 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-tls-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128610 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca5a948-2e49-4f36-8916-6502191284a6-dshm\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.128614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.128617 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3-dshm\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:24:20.800051 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.800015 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" Sep 18 20:24:20.800573 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.800055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn" event={"ID":"eca5a948-2e49-4f36-8916-6502191284a6","Type":"ContainerDied","Data":"b67bc93b448dcda7c507709614c4a09757e5182bdf9f8725c7270c05c370c27f"} Sep 18 20:24:20.800573 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.800104 2559 scope.go:117] "RemoveContainer" containerID="50710cbd22744c102c9f3221ac55be74cca5782e32a4aac7553acd644593bcd3" Sep 18 20:24:20.801968 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.801938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" event={"ID":"cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3","Type":"ContainerDied","Data":"92f3195649b16ffa8d1d026cb5d0573d6ae5dc1a1e6a7f90812a1df6cab1825c"} Sep 18 20:24:20.802071 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.801960 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl" Sep 18 20:24:20.813833 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.813809 2559 scope.go:117] "RemoveContainer" containerID="b5e7629037eec9b34531a3096f80193deaf6bd5d068c8db179e9d2236d7668a5" Sep 18 20:24:20.828641 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.828611 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn"] Sep 18 20:24:20.830606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.830583 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-w3-b88002-kserve-7c4bc9d4f6-w6gcn"] Sep 18 20:24:20.841000 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.840972 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl"] Sep 18 20:24:20.846297 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:20.846258 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b88002/distinct-2-w3-b88002-kserve-79fb6c87f-cggrl"] Sep 18 20:24:21.185316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:21.185256 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" path="/var/lib/kubelet/pods/cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3/volumes" Sep 18 20:24:21.185703 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:24:21.185689 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca5a948-2e49-4f36-8916-6502191284a6" path="/var/lib/kubelet/pods/eca5a948-2e49-4f36-8916-6502191284a6/volumes" Sep 18 20:25:14.506230 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506146 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725"] Sep 18 20:25:14.506680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506627 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca5a948-2e49-4f36-8916-6502191284a6" containerName="main" Sep 18 20:25:14.506680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506639 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca5a948-2e49-4f36-8916-6502191284a6" containerName="main" Sep 18 20:25:14.506680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506667 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce01f2e8-08a9-40dc-bc48-6326a1d536c2" containerName="payload-processing" Sep 18 20:25:14.506680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506674 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce01f2e8-08a9-40dc-bc48-6326a1d536c2" containerName="payload-processing" Sep 18 20:25:14.506680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506680 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204e9330-c547-4463-aad1-45c2daf84d1d" containerName="maas-api" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506686 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="204e9330-c547-4463-aad1-45c2daf84d1d" containerName="maas-api" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506692 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" containerName="main" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506697 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" containerName="main" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506707 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428d079a-a82c-48c1-b7c3-7cf817a5d5b3" containerName="payload-pre-processing" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506712 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="428d079a-a82c-48c1-b7c3-7cf817a5d5b3" containerName="payload-pre-processing" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506765 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="204e9330-c547-4463-aad1-45c2daf84d1d" containerName="maas-api" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506772 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6974c5-ec13-4d09-8c40-bbe6ed42b6d3" containerName="main" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506780 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce01f2e8-08a9-40dc-bc48-6326a1d536c2" containerName="payload-processing" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506786 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="428d079a-a82c-48c1-b7c3-7cf817a5d5b3" containerName="payload-pre-processing" Sep 18 20:25:14.506840 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.506792 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca5a948-2e49-4f36-8916-6502191284a6" containerName="main" Sep 18 20:25:14.517237 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.517202 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.520049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.520024 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725"] Sep 18 20:25:14.521309 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.521268 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-10f4b77d\"/\"e2e-lifecycle-model-10f4b77d-kserve-self-signed-certs\"" Sep 18 20:25:14.521309 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.521268 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-10f4b77d\"/\"default-dockercfg-8mx77\"" Sep 18 20:25:14.521494 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.521297 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-10f4b77d\"/\"kube-root-ca.crt\"" Sep 18 20:25:14.521494 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.521327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-10f4b77d\"/\"openshift-service-ca.crt\"" Sep 18 20:25:14.548021 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.547988 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-dshm\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.548212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.548034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tls-certs\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.548212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.548094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqxh\" (UniqueName: \"kubernetes.io/projected/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-kube-api-access-9fqxh\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.548212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.548176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-home\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.548212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.548204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tmp-dir\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.548392 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.548264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-model-cache\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.649169 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-model-cache\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.649169 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-dshm\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.649464 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tls-certs\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.649464 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqxh\" (UniqueName: \"kubernetes.io/projected/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-kube-api-access-9fqxh\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.649464 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-home\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.649464 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tmp-dir\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.650006 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.649977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-model-cache\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.650143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.650059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-home\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.650143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.650130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tmp-dir\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.651587 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.651566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-dshm\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.651881 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.651866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tls-certs\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.657770 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.657747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqxh\" (UniqueName: \"kubernetes.io/projected/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-kube-api-access-9fqxh\") pod \"e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:14.827491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:14.827406 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:15.101241 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:15.101216 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725"] Sep 18 20:25:15.103734 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:25:15.103706 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16cddd41_b52d_440e_aa7c_41b6c32bb5d9.slice/crio-084b733adab1807fba983ecc7b82aac1e9222b7d6f45bf36d6b87419bcf858db WatchSource:0}: Error finding container 084b733adab1807fba983ecc7b82aac1e9222b7d6f45bf36d6b87419bcf858db: Status 404 returned error can't find the container with id 084b733adab1807fba983ecc7b82aac1e9222b7d6f45bf36d6b87419bcf858db Sep 18 20:25:16.093353 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:16.093311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" event={"ID":"16cddd41-b52d-440e-aa7c-41b6c32bb5d9","Type":"ContainerStarted","Data":"1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef"} Sep 18 20:25:16.093890 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:16.093368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" event={"ID":"16cddd41-b52d-440e-aa7c-41b6c32bb5d9","Type":"ContainerStarted","Data":"084b733adab1807fba983ecc7b82aac1e9222b7d6f45bf36d6b87419bcf858db"} Sep 18 20:25:16.110185 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:16.110127 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" podStartSLOduration=2.11010836 podStartE2EDuration="2.11010836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:16.108584323 +0000 UTC m=+1435.158944772" watchObservedRunningTime="2026-09-18 20:25:16.11010836 +0000 UTC m=+1435.160468845" Sep 18 20:25:24.827910 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:24.827869 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:24.828331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:24.827939 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:24.840859 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:24.840822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:25:25.154231 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:25:25.154198 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:26:13.898415 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.898377 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597"] Sep 18 20:26:13.902049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.902024 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:13.904685 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.904662 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"openshift-service-ca.crt\"" Sep 18 20:26:13.905708 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.905688 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"default-dockercfg-lcngt\"" Sep 18 20:26:13.905857 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.905741 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"kube-root-ca.crt\"" Sep 18 20:26:13.905857 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.905819 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"auth-test-model-35558b-kserve-self-signed-certs\"" Sep 18 20:26:13.911396 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:13.911373 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597"] Sep 18 20:26:14.026267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.026228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-dshm\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.026267 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.026268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-model-cache\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.026510 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.026402 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be945ee-a861-468c-9e86-6598e51729a2-tls-certs\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.026510 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.026436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/6be945ee-a861-468c-9e86-6598e51729a2-kube-api-access-wvtt8\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.026510 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.026470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-home\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.026608 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.026546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-tmp-dir\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.127816 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.127770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-model-cache\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.127839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be945ee-a861-468c-9e86-6598e51729a2-tls-certs\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.127861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/6be945ee-a861-468c-9e86-6598e51729a2-kube-api-access-wvtt8\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.127883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-home\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.127918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-tmp-dir\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128262 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.128108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-dshm\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128262 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.128213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-model-cache\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128403 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.128312 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-home\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.128486 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.128460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-tmp-dir\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.130456 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.130428 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-dshm\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.130573 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.130530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be945ee-a861-468c-9e86-6598e51729a2-tls-certs\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.135989 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.135960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/6be945ee-a861-468c-9e86-6598e51729a2-kube-api-access-wvtt8\") pod \"auth-test-model-35558b-kserve-54659cbcf5-zq597\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.214136 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.214038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:14.486773 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:14.486633 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597"] Sep 18 20:26:14.489576 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:26:14.489549 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be945ee_a861_468c_9e86_6598e51729a2.slice/crio-1007a47049024bbeb39eef34589a5b5ba32f925b9e18229197bf0b65143e544f WatchSource:0}: Error finding container 1007a47049024bbeb39eef34589a5b5ba32f925b9e18229197bf0b65143e544f: Status 404 returned error can't find the container with id 1007a47049024bbeb39eef34589a5b5ba32f925b9e18229197bf0b65143e544f Sep 18 20:26:15.423301 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:15.423239 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" event={"ID":"6be945ee-a861-468c-9e86-6598e51729a2","Type":"ContainerStarted","Data":"24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054"} Sep 18 20:26:15.423793 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:15.423311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" event={"ID":"6be945ee-a861-468c-9e86-6598e51729a2","Type":"ContainerStarted","Data":"1007a47049024bbeb39eef34589a5b5ba32f925b9e18229197bf0b65143e544f"} Sep 18 20:26:15.440985 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:15.440934 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" podStartSLOduration=2.440917385 podStartE2EDuration="2.440917385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:15.439108478 +0000 UTC m=+1494.489468926" watchObservedRunningTime="2026-09-18 20:26:15.440917385 +0000 UTC m=+1494.491277871" Sep 18 20:26:23.239498 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:23.239457 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:26:23.241782 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:23.241755 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:26:24.214196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:24.214157 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:24.214196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:24.214207 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:24.227050 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:24.227018 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:24.479986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:24.479896 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:26:24.893224 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:24.893188 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976"] Sep 18 20:26:24.893571 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:24.893513 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" podUID="6c3ae076-606d-4fa4-b28f-55145d588cef" containerName="payload-processing" containerID="cri-o://43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905" gracePeriod=30 Sep 18 20:26:25.143867 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.143791 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:26:25.229644 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.229611 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ae076-606d-4fa4-b28f-55145d588cef-plugins-config-volume\") pod \"6c3ae076-606d-4fa4-b28f-55145d588cef\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " Sep 18 20:26:25.229846 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.229741 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd24m\" (UniqueName: \"kubernetes.io/projected/6c3ae076-606d-4fa4-b28f-55145d588cef-kube-api-access-vd24m\") pod \"6c3ae076-606d-4fa4-b28f-55145d588cef\" (UID: \"6c3ae076-606d-4fa4-b28f-55145d588cef\") " Sep 18 20:26:25.230087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.230057 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3ae076-606d-4fa4-b28f-55145d588cef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c3ae076-606d-4fa4-b28f-55145d588cef" (UID: "6c3ae076-606d-4fa4-b28f-55145d588cef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:26:25.230390 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.230369 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3ae076-606d-4fa4-b28f-55145d588cef-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:25.231934 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.231908 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3ae076-606d-4fa4-b28f-55145d588cef-kube-api-access-vd24m" (OuterVolumeSpecName: "kube-api-access-vd24m") pod "6c3ae076-606d-4fa4-b28f-55145d588cef" (UID: "6c3ae076-606d-4fa4-b28f-55145d588cef"). InnerVolumeSpecName "kube-api-access-vd24m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:26:25.331402 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.331369 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd24m\" (UniqueName: \"kubernetes.io/projected/6c3ae076-606d-4fa4-b28f-55145d588cef-kube-api-access-vd24m\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:25.475604 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.475508 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c3ae076-606d-4fa4-b28f-55145d588cef" containerID="43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905" exitCode=0 Sep 18 20:26:25.475604 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.475555 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" Sep 18 20:26:25.475796 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.475598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" event={"ID":"6c3ae076-606d-4fa4-b28f-55145d588cef","Type":"ContainerDied","Data":"43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905"} Sep 18 20:26:25.475796 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.475659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976" event={"ID":"6c3ae076-606d-4fa4-b28f-55145d588cef","Type":"ContainerDied","Data":"55579992a322f1ac18ecfb22d71b7a2f95adbc26ae1713c3079cc5758c722fb9"} Sep 18 20:26:25.475796 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.475681 2559 scope.go:117] "RemoveContainer" containerID="43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905" Sep 18 20:26:25.485180 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.485158 2559 scope.go:117] "RemoveContainer" containerID="43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905" Sep 18 20:26:25.485497 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:26:25.485464 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905\": container with ID starting with 43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905 not found: ID does not exist" containerID="43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905" Sep 18 20:26:25.485539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.485492 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905"} err="failed to get container status \"43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905\": rpc error: code = NotFound desc = could not find container \"43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905\": container with ID starting with 43b8176d7184d7834c9ba7c5fb8cca6cfca31f5c6d42df5ba62f084225b2f905 not found: ID does not exist" Sep 18 20:26:25.499214 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.499180 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976"] Sep 18 20:26:25.503666 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:25.503638 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f4b77d-6775c89ddf-jx976"] Sep 18 20:26:27.185971 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:27.185938 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3ae076-606d-4fa4-b28f-55145d588cef" path="/var/lib/kubelet/pods/6c3ae076-606d-4fa4-b28f-55145d588cef/volumes" Sep 18 20:26:33.155711 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:33.155671 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725"] Sep 18 20:26:48.175778 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.175655 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" podUID="16cddd41-b52d-440e-aa7c-41b6c32bb5d9" containerName="main" containerID="cri-o://1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef" gracePeriod=45 Sep 18 20:26:48.441658 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.441590 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:26:48.572611 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572578 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-model-cache\") pod \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " Sep 18 20:26:48.572798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572667 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-home\") pod \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " Sep 18 20:26:48.572798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572698 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fqxh\" (UniqueName: \"kubernetes.io/projected/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-kube-api-access-9fqxh\") pod \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " Sep 18 20:26:48.572798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572714 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tmp-dir\") pod \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " Sep 18 20:26:48.572798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572732 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tls-certs\") pod \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " Sep 18 20:26:48.572798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572750 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-dshm\") pod \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\" (UID: \"16cddd41-b52d-440e-aa7c-41b6c32bb5d9\") " Sep 18 20:26:48.573064 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572916 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "16cddd41-b52d-440e-aa7c-41b6c32bb5d9" (UID: "16cddd41-b52d-440e-aa7c-41b6c32bb5d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:26:48.573064 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572933 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-home" (OuterVolumeSpecName: "home") pod "16cddd41-b52d-440e-aa7c-41b6c32bb5d9" (UID: "16cddd41-b52d-440e-aa7c-41b6c32bb5d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:26:48.573064 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.572968 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16cddd41-b52d-440e-aa7c-41b6c32bb5d9" (UID: "16cddd41-b52d-440e-aa7c-41b6c32bb5d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:26:48.573225 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.573128 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-home\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:48.573225 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.573144 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tmp-dir\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:48.573225 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.573155 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-model-cache\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:48.575109 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.575082 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-dshm" (OuterVolumeSpecName: "dshm") pod "16cddd41-b52d-440e-aa7c-41b6c32bb5d9" (UID: "16cddd41-b52d-440e-aa7c-41b6c32bb5d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:26:48.575193 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.575151 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-kube-api-access-9fqxh" (OuterVolumeSpecName: "kube-api-access-9fqxh") pod "16cddd41-b52d-440e-aa7c-41b6c32bb5d9" (UID: "16cddd41-b52d-440e-aa7c-41b6c32bb5d9"). InnerVolumeSpecName "kube-api-access-9fqxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:26:48.575193 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.575179 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16cddd41-b52d-440e-aa7c-41b6c32bb5d9" (UID: "16cddd41-b52d-440e-aa7c-41b6c32bb5d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:26:48.599164 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.599135 2559 generic.go:356] "Generic (PLEG): container finished" podID="16cddd41-b52d-440e-aa7c-41b6c32bb5d9" containerID="1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef" exitCode=0 Sep 18 20:26:48.599364 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.599179 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" Sep 18 20:26:48.599364 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.599194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" event={"ID":"16cddd41-b52d-440e-aa7c-41b6c32bb5d9","Type":"ContainerDied","Data":"1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef"} Sep 18 20:26:48.599364 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.599228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725" event={"ID":"16cddd41-b52d-440e-aa7c-41b6c32bb5d9","Type":"ContainerDied","Data":"084b733adab1807fba983ecc7b82aac1e9222b7d6f45bf36d6b87419bcf858db"} Sep 18 20:26:48.599364 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.599246 2559 scope.go:117] "RemoveContainer" containerID="1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef" Sep 18 20:26:48.608979 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.608953 2559 scope.go:117] "RemoveContainer" containerID="1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef" Sep 18 20:26:48.609310 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:26:48.609283 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef\": container with ID starting with 1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef not found: ID does not exist" containerID="1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef" Sep 18 20:26:48.609414 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.609317 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef"} err="failed to get container status \"1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef\": rpc error: code = NotFound desc = could not find container \"1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef\": container with ID starting with 1ca1ee87e3f6faaae0c0d606cdb797d823112a60d1f9e395b06f406daa4230ef not found: ID does not exist" Sep 18 20:26:48.620292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.620255 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725"] Sep 18 20:26:48.624006 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.623979 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-10f4b77d/e2e-lifecycle-model-10f4b77d-kserve-5f56bf9555-mt725"] Sep 18 20:26:48.674127 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.674093 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqxh\" (UniqueName: \"kubernetes.io/projected/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-kube-api-access-9fqxh\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:48.674127 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.674122 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-tls-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:48.674127 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:48.674133 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16cddd41-b52d-440e-aa7c-41b6c32bb5d9-dshm\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:26:49.184101 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:26:49.184063 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16cddd41-b52d-440e-aa7c-41b6c32bb5d9" path="/var/lib/kubelet/pods/16cddd41-b52d-440e-aa7c-41b6c32bb5d9/volumes" Sep 18 20:27:29.003811 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.003775 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz"] Sep 18 20:27:29.004416 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.004009 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" podUID="bfe524e5-7db0-4000-b70a-2e589815d91d" containerName="maas-api" containerID="cri-o://4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1" gracePeriod=30 Sep 18 20:27:29.062209 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.062168 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz"] Sep 18 20:27:29.062491 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.062454 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" podUID="a98b5a71-2794-46a1-a3be-ac4ade059e3d" containerName="payload-processing" containerID="cri-o://2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f" gracePeriod=30 Sep 18 20:27:29.360906 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.360868 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:27:29.370349 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.370319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98b5a71-2794-46a1-a3be-ac4ade059e3d-plugins-config-volume\") pod \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " Sep 18 20:27:29.370512 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.370432 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dl9m\" (UniqueName: \"kubernetes.io/projected/a98b5a71-2794-46a1-a3be-ac4ade059e3d-kube-api-access-2dl9m\") pod \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\" (UID: \"a98b5a71-2794-46a1-a3be-ac4ade059e3d\") " Sep 18 20:27:29.371126 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.371024 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98b5a71-2794-46a1-a3be-ac4ade059e3d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a98b5a71-2794-46a1-a3be-ac4ade059e3d" (UID: "a98b5a71-2794-46a1-a3be-ac4ade059e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:27:29.373111 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.373078 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98b5a71-2794-46a1-a3be-ac4ade059e3d-kube-api-access-2dl9m" (OuterVolumeSpecName: "kube-api-access-2dl9m") pod "a98b5a71-2794-46a1-a3be-ac4ade059e3d" (UID: "a98b5a71-2794-46a1-a3be-ac4ade059e3d"). InnerVolumeSpecName "kube-api-access-2dl9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:27:29.456829 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.456803 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:27:29.471362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471333 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe524e5-7db0-4000-b70a-2e589815d91d-openshift-service-ca\") pod \"bfe524e5-7db0-4000-b70a-2e589815d91d\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " Sep 18 20:27:29.471532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471393 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-maas-api-tls\") pod \"bfe524e5-7db0-4000-b70a-2e589815d91d\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " Sep 18 20:27:29.471532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471460 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf92k\" (UniqueName: \"kubernetes.io/projected/bfe524e5-7db0-4000-b70a-2e589815d91d-kube-api-access-xf92k\") pod \"bfe524e5-7db0-4000-b70a-2e589815d91d\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " Sep 18 20:27:29.471532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471487 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-metrics-certs\") pod \"bfe524e5-7db0-4000-b70a-2e589815d91d\" (UID: \"bfe524e5-7db0-4000-b70a-2e589815d91d\") " Sep 18 20:27:29.471693 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471620 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe524e5-7db0-4000-b70a-2e589815d91d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfe524e5-7db0-4000-b70a-2e589815d91d" (UID: "bfe524e5-7db0-4000-b70a-2e589815d91d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:27:29.471869 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471844 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dl9m\" (UniqueName: \"kubernetes.io/projected/a98b5a71-2794-46a1-a3be-ac4ade059e3d-kube-api-access-2dl9m\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:27:29.471869 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471872 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98b5a71-2794-46a1-a3be-ac4ade059e3d-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:27:29.472061 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.471883 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe524e5-7db0-4000-b70a-2e589815d91d-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:27:29.473609 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.473583 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfe524e5-7db0-4000-b70a-2e589815d91d" (UID: "bfe524e5-7db0-4000-b70a-2e589815d91d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:27:29.473697 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.473620 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bfe524e5-7db0-4000-b70a-2e589815d91d" (UID: "bfe524e5-7db0-4000-b70a-2e589815d91d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:27:29.473697 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.473635 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe524e5-7db0-4000-b70a-2e589815d91d-kube-api-access-xf92k" (OuterVolumeSpecName: "kube-api-access-xf92k") pod "bfe524e5-7db0-4000-b70a-2e589815d91d" (UID: "bfe524e5-7db0-4000-b70a-2e589815d91d"). InnerVolumeSpecName "kube-api-access-xf92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:27:29.572468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.572372 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:27:29.572468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.572407 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf92k\" (UniqueName: \"kubernetes.io/projected/bfe524e5-7db0-4000-b70a-2e589815d91d-kube-api-access-xf92k\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:27:29.572468 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.572418 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe524e5-7db0-4000-b70a-2e589815d91d-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:27:29.819510 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.819476 2559 generic.go:356] "Generic (PLEG): container finished" podID="a98b5a71-2794-46a1-a3be-ac4ade059e3d" containerID="2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f" exitCode=0 Sep 18 20:27:29.819699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.819512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" event={"ID":"a98b5a71-2794-46a1-a3be-ac4ade059e3d","Type":"ContainerDied","Data":"2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f"} Sep 18 20:27:29.819699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.819544 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" Sep 18 20:27:29.819699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.819612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz" event={"ID":"a98b5a71-2794-46a1-a3be-ac4ade059e3d","Type":"ContainerDied","Data":"74ac36f768fe59234ecd5bb05d594ab1b46331b1a0786d25ec30ef77e7725974"} Sep 18 20:27:29.819699 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.819628 2559 scope.go:117] "RemoveContainer" containerID="2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f" Sep 18 20:27:29.821081 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.821058 2559 generic.go:356] "Generic (PLEG): container finished" podID="bfe524e5-7db0-4000-b70a-2e589815d91d" containerID="4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1" exitCode=0 Sep 18 20:27:29.821201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.821106 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" Sep 18 20:27:29.821201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.821115 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" event={"ID":"bfe524e5-7db0-4000-b70a-2e589815d91d","Type":"ContainerDied","Data":"4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1"} Sep 18 20:27:29.821201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.821178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz" event={"ID":"bfe524e5-7db0-4000-b70a-2e589815d91d","Type":"ContainerDied","Data":"8ac1eee13442647093dfb8b31363b4b8201d3124e60074345f511d0fdbb27893"} Sep 18 20:27:29.829798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.829776 2559 scope.go:117] "RemoveContainer" containerID="2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f" Sep 18 20:27:29.830113 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:27:29.830090 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f\": container with ID starting with 2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f not found: ID does not exist" containerID="2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f" Sep 18 20:27:29.830203 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.830122 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f"} err="failed to get container status \"2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f\": rpc error: code = NotFound desc = could not find container \"2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f\": container with ID starting with 2a31833330d537b2f70c252d36a8340e0bca19828da3d9570bca5322f0348d8f not found: ID does not exist" Sep 18 20:27:29.830203 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.830142 2559 scope.go:117] "RemoveContainer" containerID="4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1" Sep 18 20:27:29.840999 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.840981 2559 scope.go:117] "RemoveContainer" containerID="4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1" Sep 18 20:27:29.841297 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:27:29.841261 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1\": container with ID starting with 4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1 not found: ID does not exist" containerID="4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1" Sep 18 20:27:29.841345 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.841304 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1"} err="failed to get container status \"4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1\": rpc error: code = NotFound desc = could not find container \"4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1\": container with ID starting with 4666bb9b5f5b9a7ad7d634a7b3d0dc504ba2f703e71af01194a9b588fcf114f1 not found: ID does not exist" Sep 18 20:27:29.844811 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.844786 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz"] Sep 18 20:27:29.848728 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.848705 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-116778-7f674fd8c5-49snz"] Sep 18 20:27:29.861286 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.861247 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz"] Sep 18 20:27:29.865126 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:29.865103 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-116778-647c6f8d88-wjbsz"] Sep 18 20:27:31.183619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:31.183588 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98b5a71-2794-46a1-a3be-ac4ade059e3d" path="/var/lib/kubelet/pods/a98b5a71-2794-46a1-a3be-ac4ade059e3d/volumes" Sep 18 20:27:31.184011 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:27:31.183961 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe524e5-7db0-4000-b70a-2e589815d91d" path="/var/lib/kubelet/pods/bfe524e5-7db0-4000-b70a-2e589815d91d/volumes" Sep 18 20:28:16.804332 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804216 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8"] Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804689 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3ae076-606d-4fa4-b28f-55145d588cef" containerName="payload-processing" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804705 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3ae076-606d-4fa4-b28f-55145d588cef" containerName="payload-processing" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804726 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe524e5-7db0-4000-b70a-2e589815d91d" containerName="maas-api" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804731 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe524e5-7db0-4000-b70a-2e589815d91d" containerName="maas-api" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804739 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16cddd41-b52d-440e-aa7c-41b6c32bb5d9" containerName="main" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804745 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16cddd41-b52d-440e-aa7c-41b6c32bb5d9" containerName="main" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804757 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98b5a71-2794-46a1-a3be-ac4ade059e3d" containerName="payload-processing" Sep 18 20:28:16.804777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804764 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98b5a71-2794-46a1-a3be-ac4ade059e3d" containerName="payload-processing" Sep 18 20:28:16.805026 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804826 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3ae076-606d-4fa4-b28f-55145d588cef" containerName="payload-processing" Sep 18 20:28:16.805026 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804833 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe524e5-7db0-4000-b70a-2e589815d91d" containerName="maas-api" Sep 18 20:28:16.805026 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804842 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98b5a71-2794-46a1-a3be-ac4ade059e3d" containerName="payload-processing" Sep 18 20:28:16.805026 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.804848 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16cddd41-b52d-440e-aa7c-41b6c32bb5d9" containerName="main" Sep 18 20:28:16.807857 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.807834 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.810427 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.810399 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-77f5c1-serving-cert\"" Sep 18 20:28:16.817926 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.817905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8"] Sep 18 20:28:16.821316 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.821258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlr25\" (UniqueName: \"kubernetes.io/projected/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-kube-api-access-zlr25\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.821434 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.821367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.821496 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.821480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-metrics-certs\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.821561 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.821539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-openshift-service-ca\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.923244 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.923203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlr25\" (UniqueName: \"kubernetes.io/projected/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-kube-api-access-zlr25\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.923244 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.923249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.923528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.923338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-metrics-certs\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.923528 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.923389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-openshift-service-ca\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.923528 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:28:16.923449 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-serving-cert: secret "maas-api-e2e-api-a-77f5c1-serving-cert" not found Sep 18 20:28:16.923528 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:28:16.923522 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls podName:e2613dcf-4b55-4ad9-873b-daf5c9fd7b11 nodeName:}" failed. No retries permitted until 2026-09-18 20:28:17.423500716 +0000 UTC m=+1616.473861143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls") pod "maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" (UID: "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11") : secret "maas-api-e2e-api-a-77f5c1-serving-cert" not found Sep 18 20:28:16.924162 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.924132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-openshift-service-ca\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.925955 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.925932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-metrics-certs\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:16.931370 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:16.931343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlr25\" (UniqueName: \"kubernetes.io/projected/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-kube-api-access-zlr25\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:17.378928 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.378894 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7"] Sep 18 20:28:17.382671 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.382652 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.385550 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.385523 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-77f5c1-dockercfg-sgkqc\"" Sep 18 20:28:17.385550 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.385547 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-77f5c1\"" Sep 18 20:28:17.393019 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.392993 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7"] Sep 18 20:28:17.417042 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.417002 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq"] Sep 18 20:28:17.420534 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.420517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.428320 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.428266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzr6\" (UniqueName: \"kubernetes.io/projected/acdfc0f8-be56-400d-a303-90211e858d89-kube-api-access-5vzr6\") pod \"payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.428577 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.428542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdfc0f8-be56-400d-a303-90211e858d89-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.428688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.428605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:17.430334 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.430313 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq"] Sep 18 20:28:17.431339 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.431318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls\") pod \"maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:17.529355 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.529319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdfc0f8-be56-400d-a303-90211e858d89-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.529547 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.529390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjr2v\" (UniqueName: \"kubernetes.io/projected/3c615398-c609-4689-bec7-00ae3ca17d66-kube-api-access-mjr2v\") pod \"payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.529547 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.529475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c615398-c609-4689-bec7-00ae3ca17d66-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.529657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.529572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzr6\" (UniqueName: \"kubernetes.io/projected/acdfc0f8-be56-400d-a303-90211e858d89-kube-api-access-5vzr6\") pod \"payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.530101 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.530082 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdfc0f8-be56-400d-a303-90211e858d89-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.538834 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.538811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzr6\" (UniqueName: \"kubernetes.io/projected/acdfc0f8-be56-400d-a303-90211e858d89-kube-api-access-5vzr6\") pod \"payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.630532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.630438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjr2v\" (UniqueName: \"kubernetes.io/projected/3c615398-c609-4689-bec7-00ae3ca17d66-kube-api-access-mjr2v\") pod \"payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.630532 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.630487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c615398-c609-4689-bec7-00ae3ca17d66-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.631180 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.631153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c615398-c609-4689-bec7-00ae3ca17d66-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.638745 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.638714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjr2v\" (UniqueName: \"kubernetes.io/projected/3c615398-c609-4689-bec7-00ae3ca17d66-kube-api-access-mjr2v\") pod \"payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.695112 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.695065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:17.720227 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.720193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:17.731897 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.731870 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:17.971062 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:17.971038 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7"] Sep 18 20:28:17.972920 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:28:17.972891 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdfc0f8_be56_400d_a303_90211e858d89.slice/crio-fd7e6c943a2e58dda4b5d05be0f2e2c15394be31536a15719f5eafbdd234e6cc WatchSource:0}: Error finding container fd7e6c943a2e58dda4b5d05be0f2e2c15394be31536a15719f5eafbdd234e6cc: Status 404 returned error can't find the container with id fd7e6c943a2e58dda4b5d05be0f2e2c15394be31536a15719f5eafbdd234e6cc Sep 18 20:28:18.011875 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.011777 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8"] Sep 18 20:28:18.021575 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:28:18.021540 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2613dcf_4b55_4ad9_873b_daf5c9fd7b11.slice/crio-43165f4a1a4fd05d0fd3d8433c8a1979ce95c33a26f03591a28f79980ca9e219 WatchSource:0}: Error finding container 43165f4a1a4fd05d0fd3d8433c8a1979ce95c33a26f03591a28f79980ca9e219: Status 404 returned error can't find the container with id 43165f4a1a4fd05d0fd3d8433c8a1979ce95c33a26f03591a28f79980ca9e219 Sep 18 20:28:18.023420 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.023400 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:28:18.032785 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.032757 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq"] Sep 18 20:28:18.034749 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:28:18.034723 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c615398_c609_4689_bec7_00ae3ca17d66.slice/crio-990f7ea2646358c10c1a85434ceb2585ea984dd5bd436ccf7b0de6bba3b1b3f4 WatchSource:0}: Error finding container 990f7ea2646358c10c1a85434ceb2585ea984dd5bd436ccf7b0de6bba3b1b3f4: Status 404 returned error can't find the container with id 990f7ea2646358c10c1a85434ceb2585ea984dd5bd436ccf7b0de6bba3b1b3f4 Sep 18 20:28:18.080495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.080448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" event={"ID":"3c615398-c609-4689-bec7-00ae3ca17d66","Type":"ContainerStarted","Data":"990f7ea2646358c10c1a85434ceb2585ea984dd5bd436ccf7b0de6bba3b1b3f4"} Sep 18 20:28:18.082398 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.082309 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" event={"ID":"acdfc0f8-be56-400d-a303-90211e858d89","Type":"ContainerStarted","Data":"fa344dd3a686b292e56d92bb8aae4cb126b4f953b202d9cfe2aed44c20f350b4"} Sep 18 20:28:18.082398 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.082372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" event={"ID":"acdfc0f8-be56-400d-a303-90211e858d89","Type":"ContainerStarted","Data":"fd7e6c943a2e58dda4b5d05be0f2e2c15394be31536a15719f5eafbdd234e6cc"} Sep 18 20:28:18.082564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.082403 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:18.083565 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.083545 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" event={"ID":"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11","Type":"ContainerStarted","Data":"43165f4a1a4fd05d0fd3d8433c8a1979ce95c33a26f03591a28f79980ca9e219"} Sep 18 20:28:18.098914 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:18.098853 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" podStartSLOduration=1.09883016 podStartE2EDuration="1.09883016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:18.097979092 +0000 UTC m=+1617.148339552" watchObservedRunningTime="2026-09-18 20:28:18.09883016 +0000 UTC m=+1617.149190609" Sep 18 20:28:19.090723 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:19.090687 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" event={"ID":"3c615398-c609-4689-bec7-00ae3ca17d66","Type":"ContainerStarted","Data":"1e15f1d191e4c5e31d1d1125be32c167377b581f3e9067cb9b64b9d5067ff9dc"} Sep 18 20:28:19.091219 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:19.090759 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:19.092223 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:19.092195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" event={"ID":"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11","Type":"ContainerStarted","Data":"62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18"} Sep 18 20:28:19.107613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:19.107567 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" podStartSLOduration=2.107551977 podStartE2EDuration="2.107551977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:19.106813067 +0000 UTC m=+1618.157173517" watchObservedRunningTime="2026-09-18 20:28:19.107551977 +0000 UTC m=+1618.157912464" Sep 18 20:28:19.124850 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:19.124796 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" podStartSLOduration=2.773677514 podStartE2EDuration="3.124778513s" totalImagesPullingTime="351.100999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:19.122250269 +0000 UTC m=+1618.172610718" watchObservedRunningTime="2026-09-18 20:28:19.124778513 +0000 UTC m=+1618.175138960" Sep 18 20:28:20.097942 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:20.097910 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:26.110088 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:26.110054 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:28:29.095063 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:29.095022 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:28:30.100453 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:30.100425 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:28:52.019098 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.019056 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58"] Sep 18 20:28:52.023412 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.023392 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.026367 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.026338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-102c83-dockercfg-4ttqk\"" Sep 18 20:28:52.026538 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.026384 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-102c83\"" Sep 18 20:28:52.031957 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.031929 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58"] Sep 18 20:28:52.076682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.076648 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcv98\" (UniqueName: \"kubernetes.io/projected/45723f3e-d120-46b7-a071-dcdfaa851041-kube-api-access-lcv98\") pod \"payload-processing-e2e-api-b-102c83-84d4578867-l2r58\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.076870 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.076799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45723f3e-d120-46b7-a071-dcdfaa851041-plugins-config-volume\") pod \"payload-processing-e2e-api-b-102c83-84d4578867-l2r58\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.178149 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.178106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45723f3e-d120-46b7-a071-dcdfaa851041-plugins-config-volume\") pod \"payload-processing-e2e-api-b-102c83-84d4578867-l2r58\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.178383 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.178201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcv98\" (UniqueName: \"kubernetes.io/projected/45723f3e-d120-46b7-a071-dcdfaa851041-kube-api-access-lcv98\") pod \"payload-processing-e2e-api-b-102c83-84d4578867-l2r58\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.178915 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.178891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45723f3e-d120-46b7-a071-dcdfaa851041-plugins-config-volume\") pod \"payload-processing-e2e-api-b-102c83-84d4578867-l2r58\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.186231 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.186204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcv98\" (UniqueName: \"kubernetes.io/projected/45723f3e-d120-46b7-a071-dcdfaa851041-kube-api-access-lcv98\") pod \"payload-processing-e2e-api-b-102c83-84d4578867-l2r58\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.336350 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.336243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:52.628332 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:52.628302 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58"] Sep 18 20:28:52.630074 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:28:52.630041 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45723f3e_d120_46b7_a071_dcdfaa851041.slice/crio-ca837d2e5ad02243e79f4714261fe5127efa1bacaf45e98ef1150f28903696ca WatchSource:0}: Error finding container ca837d2e5ad02243e79f4714261fe5127efa1bacaf45e98ef1150f28903696ca: Status 404 returned error can't find the container with id ca837d2e5ad02243e79f4714261fe5127efa1bacaf45e98ef1150f28903696ca Sep 18 20:28:53.284580 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:53.284538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" event={"ID":"45723f3e-d120-46b7-a071-dcdfaa851041","Type":"ContainerStarted","Data":"0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f"} Sep 18 20:28:53.285049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:53.284605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" event={"ID":"45723f3e-d120-46b7-a071-dcdfaa851041","Type":"ContainerStarted","Data":"ca837d2e5ad02243e79f4714261fe5127efa1bacaf45e98ef1150f28903696ca"} Sep 18 20:28:53.285049 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:53.284760 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:28:53.301254 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:28:53.301195 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" podStartSLOduration=2.301170095 podStartE2EDuration="2.301170095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:53.300216116 +0000 UTC m=+1652.350576564" watchObservedRunningTime="2026-09-18 20:28:53.301170095 +0000 UTC m=+1652.351530548" Sep 18 20:29:04.292495 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:04.292459 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:29:07.879466 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:07.879428 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597"] Sep 18 20:29:22.900895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:22.900828 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" podUID="6be945ee-a861-468c-9e86-6598e51729a2" containerName="main" containerID="cri-o://24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054" gracePeriod=45 Sep 18 20:29:23.170245 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.170207 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:29:23.268457 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268423 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-model-cache\") pod \"6be945ee-a861-468c-9e86-6598e51729a2\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " Sep 18 20:29:23.268682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268508 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be945ee-a861-468c-9e86-6598e51729a2-tls-certs\") pod \"6be945ee-a861-468c-9e86-6598e51729a2\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " Sep 18 20:29:23.268682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268554 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-tmp-dir\") pod \"6be945ee-a861-468c-9e86-6598e51729a2\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " Sep 18 20:29:23.268682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268596 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-dshm\") pod \"6be945ee-a861-468c-9e86-6598e51729a2\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " Sep 18 20:29:23.268682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268629 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-home\") pod \"6be945ee-a861-468c-9e86-6598e51729a2\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " Sep 18 20:29:23.268682 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268658 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/6be945ee-a861-468c-9e86-6598e51729a2-kube-api-access-wvtt8\") pod \"6be945ee-a861-468c-9e86-6598e51729a2\" (UID: \"6be945ee-a861-468c-9e86-6598e51729a2\") " Sep 18 20:29:23.268966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268831 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6be945ee-a861-468c-9e86-6598e51729a2" (UID: "6be945ee-a861-468c-9e86-6598e51729a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:29:23.268966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.268875 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6be945ee-a861-468c-9e86-6598e51729a2" (UID: "6be945ee-a861-468c-9e86-6598e51729a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:29:23.269066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.269011 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-tmp-dir\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:23.269066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.269026 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-model-cache\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:23.269132 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.269067 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-home" (OuterVolumeSpecName: "home") pod "6be945ee-a861-468c-9e86-6598e51729a2" (UID: "6be945ee-a861-468c-9e86-6598e51729a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:29:23.270891 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.270865 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-dshm" (OuterVolumeSpecName: "dshm") pod "6be945ee-a861-468c-9e86-6598e51729a2" (UID: "6be945ee-a861-468c-9e86-6598e51729a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:29:23.271009 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.270926 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be945ee-a861-468c-9e86-6598e51729a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6be945ee-a861-468c-9e86-6598e51729a2" (UID: "6be945ee-a861-468c-9e86-6598e51729a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:29:23.271009 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.270938 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be945ee-a861-468c-9e86-6598e51729a2-kube-api-access-wvtt8" (OuterVolumeSpecName: "kube-api-access-wvtt8") pod "6be945ee-a861-468c-9e86-6598e51729a2" (UID: "6be945ee-a861-468c-9e86-6598e51729a2"). InnerVolumeSpecName "kube-api-access-wvtt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:29:23.370788 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.370698 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-dshm\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:23.370788 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.370733 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be945ee-a861-468c-9e86-6598e51729a2-home\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:23.370788 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.370748 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/6be945ee-a861-468c-9e86-6598e51729a2-kube-api-access-wvtt8\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:23.370788 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.370760 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be945ee-a861-468c-9e86-6598e51729a2-tls-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:23.452216 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.452127 2559 generic.go:356] "Generic (PLEG): container finished" podID="6be945ee-a861-468c-9e86-6598e51729a2" containerID="24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054" exitCode=0 Sep 18 20:29:23.452216 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.452176 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" Sep 18 20:29:23.452459 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.452214 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" event={"ID":"6be945ee-a861-468c-9e86-6598e51729a2","Type":"ContainerDied","Data":"24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054"} Sep 18 20:29:23.452459 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.452263 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597" event={"ID":"6be945ee-a861-468c-9e86-6598e51729a2","Type":"ContainerDied","Data":"1007a47049024bbeb39eef34589a5b5ba32f925b9e18229197bf0b65143e544f"} Sep 18 20:29:23.452459 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.452308 2559 scope.go:117] "RemoveContainer" containerID="24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054" Sep 18 20:29:23.465654 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.465633 2559 scope.go:117] "RemoveContainer" containerID="24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054" Sep 18 20:29:23.466002 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:29:23.465979 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054\": container with ID starting with 24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054 not found: ID does not exist" containerID="24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054" Sep 18 20:29:23.466061 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.466012 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054"} err="failed to get container status \"24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054\": rpc error: code = NotFound desc = could not find container \"24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054\": container with ID starting with 24378c1bab0f82b1df560c0e75c5c25859abb3605c875891359c8a493c102054 not found: ID does not exist" Sep 18 20:29:23.480619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.480582 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597"] Sep 18 20:29:23.482691 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:23.482668 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/auth-test-model-35558b-kserve-54659cbcf5-zq597"] Sep 18 20:29:25.184594 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:25.184555 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be945ee-a861-468c-9e86-6598e51729a2" path="/var/lib/kubelet/pods/6be945ee-a861-468c-9e86-6598e51729a2/volumes" Sep 18 20:29:36.185687 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.185598 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8"] Sep 18 20:29:36.186442 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.186412 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" podUID="e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" containerName="maas-api" containerID="cri-o://62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18" gracePeriod=30 Sep 18 20:29:36.238800 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.238756 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq"] Sep 18 20:29:36.239841 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.239793 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" podUID="3c615398-c609-4689-bec7-00ae3ca17d66" containerName="payload-processing" containerID="cri-o://1e15f1d191e4c5e31d1d1125be32c167377b581f3e9067cb9b64b9d5067ff9dc" gracePeriod=30 Sep 18 20:29:36.242960 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.242933 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7"] Sep 18 20:29:36.243585 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.243143 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" podUID="acdfc0f8-be56-400d-a303-90211e858d89" containerName="payload-pre-processing" containerID="cri-o://fa344dd3a686b292e56d92bb8aae4cb126b4f953b202d9cfe2aed44c20f350b4" gracePeriod=30 Sep 18 20:29:36.517369 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.517332 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:29:36.526826 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.526794 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c615398-c609-4689-bec7-00ae3ca17d66" containerID="1e15f1d191e4c5e31d1d1125be32c167377b581f3e9067cb9b64b9d5067ff9dc" exitCode=0 Sep 18 20:29:36.527012 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.526863 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" event={"ID":"3c615398-c609-4689-bec7-00ae3ca17d66","Type":"ContainerDied","Data":"1e15f1d191e4c5e31d1d1125be32c167377b581f3e9067cb9b64b9d5067ff9dc"} Sep 18 20:29:36.528787 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.528755 2559 generic.go:356] "Generic (PLEG): container finished" podID="acdfc0f8-be56-400d-a303-90211e858d89" containerID="fa344dd3a686b292e56d92bb8aae4cb126b4f953b202d9cfe2aed44c20f350b4" exitCode=0 Sep 18 20:29:36.528950 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.528822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" event={"ID":"acdfc0f8-be56-400d-a303-90211e858d89","Type":"ContainerDied","Data":"fa344dd3a686b292e56d92bb8aae4cb126b4f953b202d9cfe2aed44c20f350b4"} Sep 18 20:29:36.530430 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.530404 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" containerID="62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18" exitCode=0 Sep 18 20:29:36.530559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.530471 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" Sep 18 20:29:36.530559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.530491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" event={"ID":"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11","Type":"ContainerDied","Data":"62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18"} Sep 18 20:29:36.530559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.530530 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8" event={"ID":"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11","Type":"ContainerDied","Data":"43165f4a1a4fd05d0fd3d8433c8a1979ce95c33a26f03591a28f79980ca9e219"} Sep 18 20:29:36.530686 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.530564 2559 scope.go:117] "RemoveContainer" containerID="62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18" Sep 18 20:29:36.541767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.541745 2559 scope.go:117] "RemoveContainer" containerID="62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18" Sep 18 20:29:36.542376 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:29:36.542345 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18\": container with ID starting with 62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18 not found: ID does not exist" containerID="62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18" Sep 18 20:29:36.542501 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.542389 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18"} err="failed to get container status \"62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18\": rpc error: code = NotFound desc = could not find container \"62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18\": container with ID starting with 62d843c1a4c2c50f466b38066eec590929ae423e7e4cb50ae6631ede80ea8a18 not found: ID does not exist" Sep 18 20:29:36.570591 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.570562 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:29:36.580754 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.580728 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:29:36.593260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593229 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vzr6\" (UniqueName: \"kubernetes.io/projected/acdfc0f8-be56-400d-a303-90211e858d89-kube-api-access-5vzr6\") pod \"acdfc0f8-be56-400d-a303-90211e858d89\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " Sep 18 20:29:36.593480 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593269 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlr25\" (UniqueName: \"kubernetes.io/projected/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-kube-api-access-zlr25\") pod \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " Sep 18 20:29:36.593480 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593311 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjr2v\" (UniqueName: \"kubernetes.io/projected/3c615398-c609-4689-bec7-00ae3ca17d66-kube-api-access-mjr2v\") pod \"3c615398-c609-4689-bec7-00ae3ca17d66\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " Sep 18 20:29:36.593480 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593385 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c615398-c609-4689-bec7-00ae3ca17d66-plugins-config-volume\") pod \"3c615398-c609-4689-bec7-00ae3ca17d66\" (UID: \"3c615398-c609-4689-bec7-00ae3ca17d66\") " Sep 18 20:29:36.593480 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-openshift-service-ca\") pod \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " Sep 18 20:29:36.593480 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593449 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-metrics-certs\") pod \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " Sep 18 20:29:36.593788 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593487 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdfc0f8-be56-400d-a303-90211e858d89-plugins-config-volume\") pod \"acdfc0f8-be56-400d-a303-90211e858d89\" (UID: \"acdfc0f8-be56-400d-a303-90211e858d89\") " Sep 18 20:29:36.593788 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.593552 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls\") pod \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\" (UID: \"e2613dcf-4b55-4ad9-873b-daf5c9fd7b11\") " Sep 18 20:29:36.594054 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.594014 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdfc0f8-be56-400d-a303-90211e858d89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acdfc0f8-be56-400d-a303-90211e858d89" (UID: "acdfc0f8-be56-400d-a303-90211e858d89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:29:36.594178 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.594155 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" (UID: "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:29:36.594423 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.594351 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c615398-c609-4689-bec7-00ae3ca17d66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c615398-c609-4689-bec7-00ae3ca17d66" (UID: "3c615398-c609-4689-bec7-00ae3ca17d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:29:36.596173 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.596140 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" (UID: "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:29:36.596341 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.596164 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c615398-c609-4689-bec7-00ae3ca17d66-kube-api-access-mjr2v" (OuterVolumeSpecName: "kube-api-access-mjr2v") pod "3c615398-c609-4689-bec7-00ae3ca17d66" (UID: "3c615398-c609-4689-bec7-00ae3ca17d66"). InnerVolumeSpecName "kube-api-access-mjr2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:29:36.596341 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.596202 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-kube-api-access-zlr25" (OuterVolumeSpecName: "kube-api-access-zlr25") pod "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" (UID: "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11"). InnerVolumeSpecName "kube-api-access-zlr25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:29:36.596516 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.596492 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdfc0f8-be56-400d-a303-90211e858d89-kube-api-access-5vzr6" (OuterVolumeSpecName: "kube-api-access-5vzr6") pod "acdfc0f8-be56-400d-a303-90211e858d89" (UID: "acdfc0f8-be56-400d-a303-90211e858d89"). InnerVolumeSpecName "kube-api-access-5vzr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:29:36.596619 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.596599 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" (UID: "e2613dcf-4b55-4ad9-873b-daf5c9fd7b11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:29:36.694236 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694192 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c615398-c609-4689-bec7-00ae3ca17d66-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694236 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694229 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694236 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694239 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694248 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdfc0f8-be56-400d-a303-90211e858d89-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694259 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694267 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vzr6\" (UniqueName: \"kubernetes.io/projected/acdfc0f8-be56-400d-a303-90211e858d89-kube-api-access-5vzr6\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694305 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlr25\" (UniqueName: \"kubernetes.io/projected/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11-kube-api-access-zlr25\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.694527 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.694315 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjr2v\" (UniqueName: \"kubernetes.io/projected/3c615398-c609-4689-bec7-00ae3ca17d66-kube-api-access-mjr2v\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:36.852974 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.852942 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8"] Sep 18 20:29:36.856952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:36.856899 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77f5c1-5f6df7bd87-zphv8"] Sep 18 20:29:37.185566 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.185529 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" path="/var/lib/kubelet/pods/e2613dcf-4b55-4ad9-873b-daf5c9fd7b11/volumes" Sep 18 20:29:37.539071 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.538981 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" Sep 18 20:29:37.539071 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.538981 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq" event={"ID":"3c615398-c609-4689-bec7-00ae3ca17d66","Type":"ContainerDied","Data":"990f7ea2646358c10c1a85434ceb2585ea984dd5bd436ccf7b0de6bba3b1b3f4"} Sep 18 20:29:37.539628 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.539117 2559 scope.go:117] "RemoveContainer" containerID="1e15f1d191e4c5e31d1d1125be32c167377b581f3e9067cb9b64b9d5067ff9dc" Sep 18 20:29:37.540919 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.540895 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" Sep 18 20:29:37.540996 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.540935 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7" event={"ID":"acdfc0f8-be56-400d-a303-90211e858d89","Type":"ContainerDied","Data":"fd7e6c943a2e58dda4b5d05be0f2e2c15394be31536a15719f5eafbdd234e6cc"} Sep 18 20:29:37.549073 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.549049 2559 scope.go:117] "RemoveContainer" containerID="fa344dd3a686b292e56d92bb8aae4cb126b4f953b202d9cfe2aed44c20f350b4" Sep 18 20:29:37.558697 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.558671 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq"] Sep 18 20:29:37.562820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.562791 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77f5c1-54fd9b8d7c-5dvtq"] Sep 18 20:29:37.573118 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.573083 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7"] Sep 18 20:29:37.576691 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:37.576658 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77f5c1-77c8b8c5b6-k6qs7"] Sep 18 20:29:39.184875 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:39.184828 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c615398-c609-4689-bec7-00ae3ca17d66" path="/var/lib/kubelet/pods/3c615398-c609-4689-bec7-00ae3ca17d66/volumes" Sep 18 20:29:39.185422 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:39.185372 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acdfc0f8-be56-400d-a303-90211e858d89" path="/var/lib/kubelet/pods/acdfc0f8-be56-400d-a303-90211e858d89/volumes" Sep 18 20:29:58.656252 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:58.656215 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58"] Sep 18 20:29:58.656887 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:58.656488 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" podUID="45723f3e-d120-46b7-a071-dcdfaa851041" containerName="payload-processing" containerID="cri-o://0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f" gracePeriod=30 Sep 18 20:29:58.926779 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:58.926750 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:29:59.023566 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.023527 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45723f3e-d120-46b7-a071-dcdfaa851041-plugins-config-volume\") pod \"45723f3e-d120-46b7-a071-dcdfaa851041\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " Sep 18 20:29:59.023566 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.023573 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcv98\" (UniqueName: \"kubernetes.io/projected/45723f3e-d120-46b7-a071-dcdfaa851041-kube-api-access-lcv98\") pod \"45723f3e-d120-46b7-a071-dcdfaa851041\" (UID: \"45723f3e-d120-46b7-a071-dcdfaa851041\") " Sep 18 20:29:59.023937 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.023912 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45723f3e-d120-46b7-a071-dcdfaa851041-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45723f3e-d120-46b7-a071-dcdfaa851041" (UID: "45723f3e-d120-46b7-a071-dcdfaa851041"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:29:59.025774 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.025740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45723f3e-d120-46b7-a071-dcdfaa851041-kube-api-access-lcv98" (OuterVolumeSpecName: "kube-api-access-lcv98") pod "45723f3e-d120-46b7-a071-dcdfaa851041" (UID: "45723f3e-d120-46b7-a071-dcdfaa851041"). InnerVolumeSpecName "kube-api-access-lcv98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:29:59.125321 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.125253 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45723f3e-d120-46b7-a071-dcdfaa851041-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:59.125321 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.125317 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcv98\" (UniqueName: \"kubernetes.io/projected/45723f3e-d120-46b7-a071-dcdfaa851041-kube-api-access-lcv98\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:29:59.664702 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.664660 2559 generic.go:356] "Generic (PLEG): container finished" podID="45723f3e-d120-46b7-a071-dcdfaa851041" containerID="0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f" exitCode=0 Sep 18 20:29:59.665242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.664714 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" Sep 18 20:29:59.665242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.664727 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" event={"ID":"45723f3e-d120-46b7-a071-dcdfaa851041","Type":"ContainerDied","Data":"0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f"} Sep 18 20:29:59.665242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.664780 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58" event={"ID":"45723f3e-d120-46b7-a071-dcdfaa851041","Type":"ContainerDied","Data":"ca837d2e5ad02243e79f4714261fe5127efa1bacaf45e98ef1150f28903696ca"} Sep 18 20:29:59.665242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.664806 2559 scope.go:117] "RemoveContainer" containerID="0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f" Sep 18 20:29:59.674590 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.674569 2559 scope.go:117] "RemoveContainer" containerID="0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f" Sep 18 20:29:59.674928 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:29:59.674905 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f\": container with ID starting with 0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f not found: ID does not exist" containerID="0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f" Sep 18 20:29:59.674968 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.674939 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f"} err="failed to get container status \"0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f\": rpc error: code = NotFound desc = could not find container \"0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f\": container with ID starting with 0123a050406f0124f216eb54386713ee29d5d17a22ead03b05c652011d06db4f not found: ID does not exist" Sep 18 20:29:59.682030 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.681988 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58"] Sep 18 20:29:59.691515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:29:59.691476 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-102c83-84d4578867-l2r58"] Sep 18 20:30:00.134905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.134867 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l"] Sep 18 20:30:00.135545 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135517 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acdfc0f8-be56-400d-a303-90211e858d89" containerName="payload-pre-processing" Sep 18 20:30:00.135545 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135544 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="acdfc0f8-be56-400d-a303-90211e858d89" containerName="payload-pre-processing" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135556 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c615398-c609-4689-bec7-00ae3ca17d66" containerName="payload-processing" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135564 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c615398-c609-4689-bec7-00ae3ca17d66" containerName="payload-processing" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135577 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45723f3e-d120-46b7-a071-dcdfaa851041" containerName="payload-processing" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135586 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45723f3e-d120-46b7-a071-dcdfaa851041" containerName="payload-processing" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135606 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be945ee-a861-468c-9e86-6598e51729a2" containerName="main" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135614 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be945ee-a861-468c-9e86-6598e51729a2" containerName="main" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135626 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" containerName="maas-api" Sep 18 20:30:00.135715 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135633 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" containerName="maas-api" Sep 18 20:30:00.136022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135720 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="acdfc0f8-be56-400d-a303-90211e858d89" containerName="payload-pre-processing" Sep 18 20:30:00.136022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135734 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be945ee-a861-468c-9e86-6598e51729a2" containerName="main" Sep 18 20:30:00.136022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135742 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c615398-c609-4689-bec7-00ae3ca17d66" containerName="payload-processing" Sep 18 20:30:00.136022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135753 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2613dcf-4b55-4ad9-873b-daf5c9fd7b11" containerName="maas-api" Sep 18 20:30:00.136022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.135763 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45723f3e-d120-46b7-a071-dcdfaa851041" containerName="payload-processing" Sep 18 20:30:00.140335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.140309 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7"] Sep 18 20:30:00.140462 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.140441 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" Sep 18 20:30:00.143337 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.143308 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5nwx7\"" Sep 18 20:30:00.144931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.144898 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" Sep 18 20:30:00.153829 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.152828 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l"] Sep 18 20:30:00.164915 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.164883 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7"] Sep 18 20:30:00.170091 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.170061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" Sep 18 20:30:00.177108 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.177078 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" Sep 18 20:30:00.460335 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.460303 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l"] Sep 18 20:30:00.462815 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:30:00.462779 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395baf0a_f286_4da8_986e_f74c30172625.slice/crio-40cc1e9c443ca14115a6a3b9912478b0d3f6091954c4ee4d31b805114abad4fd WatchSource:0}: Error finding container 40cc1e9c443ca14115a6a3b9912478b0d3f6091954c4ee4d31b805114abad4fd: Status 404 returned error can't find the container with id 40cc1e9c443ca14115a6a3b9912478b0d3f6091954c4ee4d31b805114abad4fd Sep 18 20:30:00.469084 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.469061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7"] Sep 18 20:30:00.472342 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:30:00.472312 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a85102_6b8b_49bf_b1df_b6a62534ffa6.slice/crio-4efd9b7568c43c20c0a18579889b645ea1bdcb5b24da4639039cb1eab82027c9 WatchSource:0}: Error finding container 4efd9b7568c43c20c0a18579889b645ea1bdcb5b24da4639039cb1eab82027c9: Status 404 returned error can't find the container with id 4efd9b7568c43c20c0a18579889b645ea1bdcb5b24da4639039cb1eab82027c9 Sep 18 20:30:00.673033 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.672995 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" event={"ID":"e7a85102-6b8b-49bf-b1df-b6a62534ffa6","Type":"ContainerStarted","Data":"4efd9b7568c43c20c0a18579889b645ea1bdcb5b24da4639039cb1eab82027c9"} Sep 18 20:30:00.674266 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:00.674234 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" event={"ID":"395baf0a-f286-4da8-986e-f74c30172625","Type":"ContainerStarted","Data":"40cc1e9c443ca14115a6a3b9912478b0d3f6091954c4ee4d31b805114abad4fd"} Sep 18 20:30:01.185000 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:01.184962 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45723f3e-d120-46b7-a071-dcdfaa851041" path="/var/lib/kubelet/pods/45723f3e-d120-46b7-a071-dcdfaa851041/volumes" Sep 18 20:30:03.696233 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:03.696190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" event={"ID":"e7a85102-6b8b-49bf-b1df-b6a62534ffa6","Type":"ContainerStarted","Data":"6a6ef16637193d61d9b42d17de6fea40fdfe47a8c8348211e0de55d5733257d4"} Sep 18 20:30:03.698136 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:03.698105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" event={"ID":"395baf0a-f286-4da8-986e-f74c30172625","Type":"ContainerStarted","Data":"20afe2cb5b58bd1babb83358c742da4bf9317295b9381c66235edd6410b588c0"} Sep 18 20:30:03.714772 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:03.714713 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" podStartSLOduration=1.404629745 podStartE2EDuration="3.714695374s" totalImagesPullingTime="2.310065629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:03.711863008 +0000 UTC m=+1722.762223455" watchObservedRunningTime="2026-09-18 20:30:03.714695374 +0000 UTC m=+1722.765055818" Sep 18 20:30:03.731928 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:03.731864 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" podStartSLOduration=1.407321919 podStartE2EDuration="3.731846339s" totalImagesPullingTime="2.32452442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:03.729418801 +0000 UTC m=+1722.779779257" watchObservedRunningTime="2026-09-18 20:30:03.731846339 +0000 UTC m=+1722.782206787" Sep 18 20:30:04.705678 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:04.705642 2559 generic.go:356] "Generic (PLEG): container finished" podID="e7a85102-6b8b-49bf-b1df-b6a62534ffa6" containerID="6a6ef16637193d61d9b42d17de6fea40fdfe47a8c8348211e0de55d5733257d4" exitCode=0 Sep 18 20:30:04.706196 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:04.705728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" event={"ID":"e7a85102-6b8b-49bf-b1df-b6a62534ffa6","Type":"ContainerDied","Data":"6a6ef16637193d61d9b42d17de6fea40fdfe47a8c8348211e0de55d5733257d4"} Sep 18 20:30:04.707347 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:04.707316 2559 generic.go:356] "Generic (PLEG): container finished" podID="395baf0a-f286-4da8-986e-f74c30172625" containerID="20afe2cb5b58bd1babb83358c742da4bf9317295b9381c66235edd6410b588c0" exitCode=0 Sep 18 20:30:04.707480 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:04.707362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" event={"ID":"395baf0a-f286-4da8-986e-f74c30172625","Type":"ContainerDied","Data":"20afe2cb5b58bd1babb83358c742da4bf9317295b9381c66235edd6410b588c0"} Sep 18 20:30:05.867539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:05.867504 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" Sep 18 20:30:05.891168 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:05.891140 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" Sep 18 20:30:06.721711 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:06.721666 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" Sep 18 20:30:06.721711 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:06.721663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7" event={"ID":"e7a85102-6b8b-49bf-b1df-b6a62534ffa6","Type":"ContainerDied","Data":"4efd9b7568c43c20c0a18579889b645ea1bdcb5b24da4639039cb1eab82027c9"} Sep 18 20:30:06.722067 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:06.721746 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4efd9b7568c43c20c0a18579889b645ea1bdcb5b24da4639039cb1eab82027c9" Sep 18 20:30:06.723212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:06.723187 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" Sep 18 20:30:06.723336 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:06.723204 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l" event={"ID":"395baf0a-f286-4da8-986e-f74c30172625","Type":"ContainerDied","Data":"40cc1e9c443ca14115a6a3b9912478b0d3f6091954c4ee4d31b805114abad4fd"} Sep 18 20:30:06.723336 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:06.723265 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40cc1e9c443ca14115a6a3b9912478b0d3f6091954c4ee4d31b805114abad4fd" Sep 18 20:30:16.182342 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.182304 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h"] Sep 18 20:30:16.182904 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.182538 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" podUID="faca8468-ffba-4da3-857a-50ee50360016" containerName="maas-api" containerID="cri-o://d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e" gracePeriod=30 Sep 18 20:30:16.214128 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.214074 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l"] Sep 18 20:30:16.215995 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.215967 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35f893-29829390-ldw9l"] Sep 18 20:30:16.438075 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.438001 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:30:16.486470 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.486433 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faca8468-ffba-4da3-857a-50ee50360016-openshift-service-ca\") pod \"faca8468-ffba-4da3-857a-50ee50360016\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " Sep 18 20:30:16.486669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.486528 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnffk\" (UniqueName: \"kubernetes.io/projected/faca8468-ffba-4da3-857a-50ee50360016-kube-api-access-tnffk\") pod \"faca8468-ffba-4da3-857a-50ee50360016\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " Sep 18 20:30:16.486669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.486572 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-metrics-certs\") pod \"faca8468-ffba-4da3-857a-50ee50360016\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " Sep 18 20:30:16.486669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.486619 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls\") pod \"faca8468-ffba-4da3-857a-50ee50360016\" (UID: \"faca8468-ffba-4da3-857a-50ee50360016\") " Sep 18 20:30:16.486879 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.486846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faca8468-ffba-4da3-857a-50ee50360016-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "faca8468-ffba-4da3-857a-50ee50360016" (UID: "faca8468-ffba-4da3-857a-50ee50360016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:30:16.488672 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.488638 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "faca8468-ffba-4da3-857a-50ee50360016" (UID: "faca8468-ffba-4da3-857a-50ee50360016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:30:16.488779 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.488703 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "faca8468-ffba-4da3-857a-50ee50360016" (UID: "faca8468-ffba-4da3-857a-50ee50360016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:30:16.488779 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.488736 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faca8468-ffba-4da3-857a-50ee50360016-kube-api-access-tnffk" (OuterVolumeSpecName: "kube-api-access-tnffk") pod "faca8468-ffba-4da3-857a-50ee50360016" (UID: "faca8468-ffba-4da3-857a-50ee50360016"). InnerVolumeSpecName "kube-api-access-tnffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:30:16.587954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.587878 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnffk\" (UniqueName: \"kubernetes.io/projected/faca8468-ffba-4da3-857a-50ee50360016-kube-api-access-tnffk\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:16.587954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.587948 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:16.587954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.587962 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faca8468-ffba-4da3-857a-50ee50360016-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:16.588217 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.588008 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faca8468-ffba-4da3-857a-50ee50360016-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:16.778086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.777983 2559 generic.go:356] "Generic (PLEG): container finished" podID="faca8468-ffba-4da3-857a-50ee50360016" containerID="d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e" exitCode=0 Sep 18 20:30:16.778086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.778033 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" Sep 18 20:30:16.778362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.778080 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" event={"ID":"faca8468-ffba-4da3-857a-50ee50360016","Type":"ContainerDied","Data":"d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e"} Sep 18 20:30:16.778362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.778158 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h" event={"ID":"faca8468-ffba-4da3-857a-50ee50360016","Type":"ContainerDied","Data":"c1409d7656cb3d47c802da21e38f639dc25e5e250f8bf6d5e3c702d45bd8cbcc"} Sep 18 20:30:16.778362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.778181 2559 scope.go:117] "RemoveContainer" containerID="d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e" Sep 18 20:30:16.788481 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.788457 2559 scope.go:117] "RemoveContainer" containerID="d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e" Sep 18 20:30:16.788799 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:16.788781 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e\": container with ID starting with d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e not found: ID does not exist" containerID="d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e" Sep 18 20:30:16.788877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.788809 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e"} err="failed to get container status \"d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e\": rpc error: code = NotFound desc = could not find container \"d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e\": container with ID starting with d4effb490410a2f10755982a1c2c5c0ebbb4d90f8c713f03684e15925ed6434e not found: ID does not exist" Sep 18 20:30:16.802904 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.802871 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h"] Sep 18 20:30:16.806994 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:16.806964 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-35f893-547bb458b8-lcd9h"] Sep 18 20:30:17.184605 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:17.184572 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395baf0a-f286-4da8-986e-f74c30172625" path="/var/lib/kubelet/pods/395baf0a-f286-4da8-986e-f74c30172625/volumes" Sep 18 20:30:17.185027 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:17.184950 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faca8468-ffba-4da3-857a-50ee50360016" path="/var/lib/kubelet/pods/faca8468-ffba-4da3-857a-50ee50360016/volumes" Sep 18 20:30:33.675612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.675576 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5"] Sep 18 20:30:33.676343 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.676310 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" podUID="eecb8a7f-2edf-4ebe-b70e-36efd74c399c" containerName="payload-processing" containerID="cri-o://86225ffbd6eccca9dab69eb2fd00d62808b6325fd642984d759f7c76f49b4ad0" gracePeriod=30 Sep 18 20:30:33.683600 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.683312 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg"] Sep 18 20:30:33.683600 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.683520 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" podUID="f4162009-fa3e-4bba-8239-8d25e81e4471" containerName="payload-pre-processing" containerID="cri-o://632f2129b2a6e0644d6a1203094f7b5f93e0b37340dc4473b98b93989229f455" gracePeriod=30 Sep 18 20:30:33.876979 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.876935 2559 generic.go:356] "Generic (PLEG): container finished" podID="eecb8a7f-2edf-4ebe-b70e-36efd74c399c" containerID="86225ffbd6eccca9dab69eb2fd00d62808b6325fd642984d759f7c76f49b4ad0" exitCode=0 Sep 18 20:30:33.877189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.876982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" event={"ID":"eecb8a7f-2edf-4ebe-b70e-36efd74c399c","Type":"ContainerDied","Data":"86225ffbd6eccca9dab69eb2fd00d62808b6325fd642984d759f7c76f49b4ad0"} Sep 18 20:30:33.879722 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.879623 2559 generic.go:356] "Generic (PLEG): container finished" podID="f4162009-fa3e-4bba-8239-8d25e81e4471" containerID="632f2129b2a6e0644d6a1203094f7b5f93e0b37340dc4473b98b93989229f455" exitCode=0 Sep 18 20:30:33.879722 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.879691 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" event={"ID":"f4162009-fa3e-4bba-8239-8d25e81e4471","Type":"ContainerDied","Data":"632f2129b2a6e0644d6a1203094f7b5f93e0b37340dc4473b98b93989229f455"} Sep 18 20:30:33.972309 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.972263 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:30:33.974350 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:33.974327 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:30:34.051759 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.051727 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-plugins-config-volume\") pod \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " Sep 18 20:30:34.051942 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.051773 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4162009-fa3e-4bba-8239-8d25e81e4471-plugins-config-volume\") pod \"f4162009-fa3e-4bba-8239-8d25e81e4471\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " Sep 18 20:30:34.051942 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.051827 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pjbh\" (UniqueName: \"kubernetes.io/projected/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-kube-api-access-4pjbh\") pod \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\" (UID: \"eecb8a7f-2edf-4ebe-b70e-36efd74c399c\") " Sep 18 20:30:34.051942 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.051843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccpxb\" (UniqueName: \"kubernetes.io/projected/f4162009-fa3e-4bba-8239-8d25e81e4471-kube-api-access-ccpxb\") pod \"f4162009-fa3e-4bba-8239-8d25e81e4471\" (UID: \"f4162009-fa3e-4bba-8239-8d25e81e4471\") " Sep 18 20:30:34.052290 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.052243 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eecb8a7f-2edf-4ebe-b70e-36efd74c399c" (UID: "eecb8a7f-2edf-4ebe-b70e-36efd74c399c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:30:34.052425 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.052400 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4162009-fa3e-4bba-8239-8d25e81e4471-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4162009-fa3e-4bba-8239-8d25e81e4471" (UID: "f4162009-fa3e-4bba-8239-8d25e81e4471"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:30:34.054151 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.054121 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4162009-fa3e-4bba-8239-8d25e81e4471-kube-api-access-ccpxb" (OuterVolumeSpecName: "kube-api-access-ccpxb") pod "f4162009-fa3e-4bba-8239-8d25e81e4471" (UID: "f4162009-fa3e-4bba-8239-8d25e81e4471"). InnerVolumeSpecName "kube-api-access-ccpxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:30:34.054260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.054155 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-kube-api-access-4pjbh" (OuterVolumeSpecName: "kube-api-access-4pjbh") pod "eecb8a7f-2edf-4ebe-b70e-36efd74c399c" (UID: "eecb8a7f-2edf-4ebe-b70e-36efd74c399c"). InnerVolumeSpecName "kube-api-access-4pjbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:30:34.152802 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.152754 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4162009-fa3e-4bba-8239-8d25e81e4471-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:34.152802 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.152796 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pjbh\" (UniqueName: \"kubernetes.io/projected/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-kube-api-access-4pjbh\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:34.152802 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.152811 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpxb\" (UniqueName: \"kubernetes.io/projected/f4162009-fa3e-4bba-8239-8d25e81e4471-kube-api-access-ccpxb\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:34.153048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.152825 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecb8a7f-2edf-4ebe-b70e-36efd74c399c-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:34.886317 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.886260 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" Sep 18 20:30:34.886317 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.886303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5" event={"ID":"eecb8a7f-2edf-4ebe-b70e-36efd74c399c","Type":"ContainerDied","Data":"de5ecb9bc0a4f1b067866378bee2d8fb794672aa7722b636439083a33ba97d8c"} Sep 18 20:30:34.886858 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.886367 2559 scope.go:117] "RemoveContainer" containerID="86225ffbd6eccca9dab69eb2fd00d62808b6325fd642984d759f7c76f49b4ad0" Sep 18 20:30:34.887902 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.887737 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" Sep 18 20:30:34.887902 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.887743 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg" event={"ID":"f4162009-fa3e-4bba-8239-8d25e81e4471","Type":"ContainerDied","Data":"5dbd30ec27ed2b006d4c36522a31cb07a26e7d827324d00ce1f408b6e9fceae6"} Sep 18 20:30:34.897034 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.897015 2559 scope.go:117] "RemoveContainer" containerID="632f2129b2a6e0644d6a1203094f7b5f93e0b37340dc4473b98b93989229f455" Sep 18 20:30:34.909661 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.909637 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5"] Sep 18 20:30:34.914370 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.914349 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9a5190-5dfbfdb79-mqcz5"] Sep 18 20:30:34.924958 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.924929 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg"] Sep 18 20:30:34.932211 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:34.932183 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9a5190-c5967f66b-9vfzg"] Sep 18 20:30:35.183543 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:35.183513 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eecb8a7f-2edf-4ebe-b70e-36efd74c399c" path="/var/lib/kubelet/pods/eecb8a7f-2edf-4ebe-b70e-36efd74c399c/volumes" Sep 18 20:30:35.183885 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:35.183872 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4162009-fa3e-4bba-8239-8d25e81e4471" path="/var/lib/kubelet/pods/f4162009-fa3e-4bba-8239-8d25e81e4471/volumes" Sep 18 20:30:37.385651 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.385612 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8719-414e5b"] Sep 18 20:30:37.386044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386012 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395baf0a-f286-4da8-986e-f74c30172625" containerName="cleanup" Sep 18 20:30:37.386044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386023 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="395baf0a-f286-4da8-986e-f74c30172625" containerName="cleanup" Sep 18 20:30:37.386044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386031 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4162009-fa3e-4bba-8239-8d25e81e4471" containerName="payload-pre-processing" Sep 18 20:30:37.386044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386037 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4162009-fa3e-4bba-8239-8d25e81e4471" containerName="payload-pre-processing" Sep 18 20:30:37.386044 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386044 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a85102-6b8b-49bf-b1df-b6a62534ffa6" containerName="cleanup" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386050 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a85102-6b8b-49bf-b1df-b6a62534ffa6" containerName="cleanup" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386067 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faca8468-ffba-4da3-857a-50ee50360016" containerName="maas-api" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386073 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="faca8468-ffba-4da3-857a-50ee50360016" containerName="maas-api" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386081 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eecb8a7f-2edf-4ebe-b70e-36efd74c399c" containerName="payload-processing" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386086 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eecb8a7f-2edf-4ebe-b70e-36efd74c399c" containerName="payload-processing" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386141 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a85102-6b8b-49bf-b1df-b6a62534ffa6" containerName="cleanup" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386148 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="faca8468-ffba-4da3-857a-50ee50360016" containerName="maas-api" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386155 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="395baf0a-f286-4da8-986e-f74c30172625" containerName="cleanup" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386163 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4162009-fa3e-4bba-8239-8d25e81e4471" containerName="payload-pre-processing" Sep 18 20:30:37.386220 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.386170 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eecb8a7f-2edf-4ebe-b70e-36efd74c399c" containerName="payload-processing" Sep 18 20:30:37.394582 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.394552 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8719-414e5b"] Sep 18 20:30:37.394731 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.394668 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:37.397247 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.397226 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pqc57\"" Sep 18 20:30:37.482806 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.482771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzts\" (UniqueName: \"kubernetes.io/projected/2ea97f7a-c231-4cc3-b73a-902332b1cad4-kube-api-access-vpzts\") pod \"test-curl-8719-414e5b\" (UID: \"2ea97f7a-c231-4cc3-b73a-902332b1cad4\") " pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:37.589707 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.589663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzts\" (UniqueName: \"kubernetes.io/projected/2ea97f7a-c231-4cc3-b73a-902332b1cad4-kube-api-access-vpzts\") pod \"test-curl-8719-414e5b\" (UID: \"2ea97f7a-c231-4cc3-b73a-902332b1cad4\") " pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:37.598890 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.598852 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzts\" (UniqueName: \"kubernetes.io/projected/2ea97f7a-c231-4cc3-b73a-902332b1cad4-kube-api-access-vpzts\") pod \"test-curl-8719-414e5b\" (UID: \"2ea97f7a-c231-4cc3-b73a-902332b1cad4\") " pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:37.704986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.704890 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:37.976538 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:37.976512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8719-414e5b"] Sep 18 20:30:37.979376 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:30:37.979347 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea97f7a_c231_4cc3_b73a_902332b1cad4.slice/crio-b2a06a0d4ae6362e75c48a1e8b4333de8b714b7080e55faa925a4ac3ea37c32c WatchSource:0}: Error finding container b2a06a0d4ae6362e75c48a1e8b4333de8b714b7080e55faa925a4ac3ea37c32c: Status 404 returned error can't find the container with id b2a06a0d4ae6362e75c48a1e8b4333de8b714b7080e55faa925a4ac3ea37c32c Sep 18 20:30:38.914269 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:38.914229 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8719-414e5b" event={"ID":"2ea97f7a-c231-4cc3-b73a-902332b1cad4","Type":"ContainerStarted","Data":"9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd"} Sep 18 20:30:38.914771 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:38.914297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8719-414e5b" event={"ID":"2ea97f7a-c231-4cc3-b73a-902332b1cad4","Type":"ContainerStarted","Data":"b2a06a0d4ae6362e75c48a1e8b4333de8b714b7080e55faa925a4ac3ea37c32c"} Sep 18 20:30:38.930463 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:38.930405 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8719-414e5b" podStartSLOduration=1.090553872 podStartE2EDuration="1.930387589s" totalImagesPullingTime="839.833717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:38.92815106 +0000 UTC m=+1757.978511508" watchObservedRunningTime="2026-09-18 20:30:38.930387589 +0000 UTC m=+1757.980748037" Sep 18 20:30:39.492935 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.492898 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8719-414e5b"] Sep 18 20:30:39.495013 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.494981 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8719-414e5b"] Sep 18 20:30:39.606433 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.606398 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8719-aea122"] Sep 18 20:30:39.606808 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.606795 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" containerName="test-curl-8719-414e5b" Sep 18 20:30:39.606854 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.606809 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" containerName="test-curl-8719-414e5b" Sep 18 20:30:39.606892 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.606867 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" containerName="test-curl-8719-414e5b" Sep 18 20:30:39.610061 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.610045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:39.615643 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.615617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8719-aea122"] Sep 18 20:30:39.713304 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.713242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpv7b\" (UniqueName: \"kubernetes.io/projected/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e-kube-api-access-mpv7b\") pod \"test-curl-8719-aea122\" (UID: \"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e\") " pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:39.813943 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.813907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpv7b\" (UniqueName: \"kubernetes.io/projected/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e-kube-api-access-mpv7b\") pod \"test-curl-8719-aea122\" (UID: \"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e\") " pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:39.822058 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.822026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpv7b\" (UniqueName: \"kubernetes.io/projected/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e-kube-api-access-mpv7b\") pod \"test-curl-8719-aea122\" (UID: \"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e\") " pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:39.919604 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:39.919515 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8719-414e5b" container="test-curl-8719-414e5b" Sep 18 20:30:39.920413 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:39.920401 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:40.196945 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:40.196866 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8719-aea122"] Sep 18 20:30:40.200736 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:30:40.200709 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1d8fad9_89df_4da4_85e2_ca9c04f9fc4e.slice/crio-5244d8571abf0b72ce0cd500bcd5834bb4a4838084af94778d4c17e2c75fbc4b WatchSource:0}: Error finding container 5244d8571abf0b72ce0cd500bcd5834bb4a4838084af94778d4c17e2c75fbc4b: Status 404 returned error can't find the container with id 5244d8571abf0b72ce0cd500bcd5834bb4a4838084af94778d4c17e2c75fbc4b Sep 18 20:30:40.932422 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:40.932383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8719-aea122" event={"ID":"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e","Type":"ContainerStarted","Data":"bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695"} Sep 18 20:30:40.932862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:40.932431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8719-aea122" event={"ID":"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e","Type":"ContainerStarted","Data":"5244d8571abf0b72ce0cd500bcd5834bb4a4838084af94778d4c17e2c75fbc4b"} Sep 18 20:30:40.932862 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:40.932508 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8719-414e5b" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" containerName="test-curl-8719-414e5b" containerID="cri-o://9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd" gracePeriod=2 Sep 18 20:30:40.935337 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:40.935311 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8719-414e5b\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:40.954113 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:40.954074 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8719-414e5b\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:40.954113 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:40.954068 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8719-aea122" podStartSLOduration=1.95405137 podStartE2EDuration="1.95405137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:40.951974723 +0000 UTC m=+1760.002335171" watchObservedRunningTime="2026-09-18 20:30:40.95405137 +0000 UTC m=+1760.004411853" Sep 18 20:30:41.523734 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:41.523695 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8719-aea122"] Sep 18 20:30:41.528204 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:41.528177 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8719-aea122"] Sep 18 20:30:41.938489 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:41.938455 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8719-aea122" container="test-curl-8719-aea122" Sep 18 20:30:42.944101 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:42.944065 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8719-aea122" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" containerName="test-curl-8719-aea122" containerID="cri-o://bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695" gracePeriod=2 Sep 18 20:30:42.946413 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:42.946382 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8719-aea122\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:43.105254 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.105230 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:43.107878 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:43.107848 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8719-aea122\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:43.181870 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:43.181830 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8719-aea122\" is forbidden: User \"system:node:ip-10-0-135-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-232.ec2.internal' and this object" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:43.244966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.244865 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpzts\" (UniqueName: \"kubernetes.io/projected/2ea97f7a-c231-4cc3-b73a-902332b1cad4-kube-api-access-vpzts\") pod \"2ea97f7a-c231-4cc3-b73a-902332b1cad4\" (UID: \"2ea97f7a-c231-4cc3-b73a-902332b1cad4\") " Sep 18 20:30:43.247066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.247031 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea97f7a-c231-4cc3-b73a-902332b1cad4-kube-api-access-vpzts" (OuterVolumeSpecName: "kube-api-access-vpzts") pod "2ea97f7a-c231-4cc3-b73a-902332b1cad4" (UID: "2ea97f7a-c231-4cc3-b73a-902332b1cad4"). InnerVolumeSpecName "kube-api-access-vpzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:30:43.346352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.346315 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzts\" (UniqueName: \"kubernetes.io/projected/2ea97f7a-c231-4cc3-b73a-902332b1cad4-kube-api-access-vpzts\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:43.950492 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.950459 2559 generic.go:356] "Generic (PLEG): container finished" podID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" containerID="9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd" exitCode=137 Sep 18 20:30:43.950920 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.950503 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-414e5b" Sep 18 20:30:43.950920 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.950564 2559 scope.go:117] "RemoveContainer" containerID="9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd" Sep 18 20:30:43.960123 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.960101 2559 scope.go:117] "RemoveContainer" containerID="9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd" Sep 18 20:30:43.960473 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:43.960455 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd\": container with ID starting with 9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd not found: ID does not exist" containerID="9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd" Sep 18 20:30:43.960539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:43.960485 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd"} err="failed to get container status \"9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd\": rpc error: code = NotFound desc = could not find container \"9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd\": container with ID starting with 9395058cb975ab1b2b95e5e87d99ec9b3ac4e36d227f818eaef54ddf92dbedfd not found: ID does not exist" Sep 18 20:30:45.137242 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.137218 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:45.183849 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.183817 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea97f7a-c231-4cc3-b73a-902332b1cad4" path="/var/lib/kubelet/pods/2ea97f7a-c231-4cc3-b73a-902332b1cad4/volumes" Sep 18 20:30:45.264346 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.264235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpv7b\" (UniqueName: \"kubernetes.io/projected/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e-kube-api-access-mpv7b\") pod \"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e\" (UID: \"e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e\") " Sep 18 20:30:45.266432 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.266401 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e-kube-api-access-mpv7b" (OuterVolumeSpecName: "kube-api-access-mpv7b") pod "e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" (UID: "e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e"). InnerVolumeSpecName "kube-api-access-mpv7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:30:45.365807 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.365768 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpv7b\" (UniqueName: \"kubernetes.io/projected/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e-kube-api-access-mpv7b\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:30:45.964010 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.963976 2559 generic.go:356] "Generic (PLEG): container finished" podID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" containerID="bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695" exitCode=137 Sep 18 20:30:45.964208 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.964017 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8719-aea122" Sep 18 20:30:45.964208 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.964078 2559 scope.go:117] "RemoveContainer" containerID="bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695" Sep 18 20:30:45.973740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.973713 2559 scope.go:117] "RemoveContainer" containerID="bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695" Sep 18 20:30:45.974079 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:30:45.974060 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695\": container with ID starting with bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695 not found: ID does not exist" containerID="bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695" Sep 18 20:30:45.974129 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:45.974090 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695"} err="failed to get container status \"bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695\": rpc error: code = NotFound desc = could not find container \"bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695\": container with ID starting with bcc6d485022c9598b1a5f91287143d4eb9aa16cc341f9064a61b6b079d066695 not found: ID does not exist" Sep 18 20:30:47.183614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:30:47.183579 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" path="/var/lib/kubelet/pods/e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e/volumes" Sep 18 20:31:23.272319 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:23.272290 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:31:23.274556 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:23.274536 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:31:35.704066 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.704031 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv"] Sep 18 20:31:35.704552 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.704459 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" containerName="test-curl-8719-aea122" Sep 18 20:31:35.704552 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.704471 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" containerName="test-curl-8719-aea122" Sep 18 20:31:35.704552 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.704540 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1d8fad9-89df-4da4-85e2-ca9c04f9fc4e" containerName="test-curl-8719-aea122" Sep 18 20:31:35.709214 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.709189 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.711752 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.711723 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"openshift-service-ca.crt\"" Sep 18 20:31:35.713159 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.713131 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"kube-root-ca.crt\"" Sep 18 20:31:35.713331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.713181 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"rate-test-model-35558b-kserve-self-signed-certs\"" Sep 18 20:31:35.713331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.713206 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-35558b\"/\"default-dockercfg-lcngt\"" Sep 18 20:31:35.716177 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.716149 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv"] Sep 18 20:31:35.742629 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.742591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-model-cache\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.742629 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.742631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57wf\" (UniqueName: \"kubernetes.io/projected/9a89ddbc-80da-423a-90e3-c73dcf03b062-kube-api-access-x57wf\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.742903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.742700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-dshm\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.742903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.742741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-tmp-dir\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.742903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.742791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a89ddbc-80da-423a-90e3-c73dcf03b062-tls-certs\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.742903 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.742816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-home\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.843533 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-model-cache\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.843533 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57wf\" (UniqueName: \"kubernetes.io/projected/9a89ddbc-80da-423a-90e3-c73dcf03b062-kube-api-access-x57wf\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.843817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-dshm\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.843817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-tmp-dir\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.843817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a89ddbc-80da-423a-90e3-c73dcf03b062-tls-certs\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.843817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843636 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-home\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.844028 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.843908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-model-cache\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.844076 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.844042 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-tmp-dir\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.844076 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.844069 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-home\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.845991 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.845957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-dshm\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.846201 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.846182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a89ddbc-80da-423a-90e3-c73dcf03b062-tls-certs\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:35.851842 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:35.851819 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57wf\" (UniqueName: \"kubernetes.io/projected/9a89ddbc-80da-423a-90e3-c73dcf03b062-kube-api-access-x57wf\") pod \"rate-test-model-35558b-kserve-54fbc87cdc-ncfnv\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:36.020797 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:36.020705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:36.292305 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:36.292262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv"] Sep 18 20:31:36.294342 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:31:36.294317 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a89ddbc_80da_423a_90e3_c73dcf03b062.slice/crio-677ec0ca5e28a8c9db38dd28a7adb806dbf116d71d5c2f2935f04e7fccbf876d WatchSource:0}: Error finding container 677ec0ca5e28a8c9db38dd28a7adb806dbf116d71d5c2f2935f04e7fccbf876d: Status 404 returned error can't find the container with id 677ec0ca5e28a8c9db38dd28a7adb806dbf116d71d5c2f2935f04e7fccbf876d Sep 18 20:31:37.222187 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:37.222152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" event={"ID":"9a89ddbc-80da-423a-90e3-c73dcf03b062","Type":"ContainerStarted","Data":"393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35"} Sep 18 20:31:37.222607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:37.222199 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" event={"ID":"9a89ddbc-80da-423a-90e3-c73dcf03b062","Type":"ContainerStarted","Data":"677ec0ca5e28a8c9db38dd28a7adb806dbf116d71d5c2f2935f04e7fccbf876d"} Sep 18 20:31:37.240690 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:37.240645 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" podStartSLOduration=2.240629429 podStartE2EDuration="2.240629429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:37.239058698 +0000 UTC m=+1816.289419172" watchObservedRunningTime="2026-09-18 20:31:37.240629429 +0000 UTC m=+1816.290989876" Sep 18 20:31:46.021420 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:46.021378 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:46.021420 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:46.021431 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:46.033950 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:46.033925 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:31:46.276891 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:31:46.276802 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:32:14.099570 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:14.099525 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv"] Sep 18 20:32:19.498223 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.498190 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz"] Sep 18 20:32:19.501950 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.501927 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.504384 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.504366 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8cdaea-serving-cert\"" Sep 18 20:32:19.510657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.510634 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz"] Sep 18 20:32:19.539905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.539875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fmz\" (UniqueName: \"kubernetes.io/projected/aeaa3d9c-d121-469c-9a55-3e57392a1855-kube-api-access-w8fmz\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.540060 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.539927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-metrics-certs\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.540060 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.539944 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.540060 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.540017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa3d9c-d121-469c-9a55-3e57392a1855-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.640553 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.640519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-metrics-certs\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.640553 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.640553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.640801 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.640587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa3d9c-d121-469c-9a55-3e57392a1855-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.640801 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.640648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fmz\" (UniqueName: \"kubernetes.io/projected/aeaa3d9c-d121-469c-9a55-3e57392a1855-kube-api-access-w8fmz\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.640801 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:32:19.640678 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-serving-cert: secret "maas-api-e2e-ipp-a-8cdaea-serving-cert" not found Sep 18 20:32:19.640801 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:32:19.640761 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls podName:aeaa3d9c-d121-469c-9a55-3e57392a1855 nodeName:}" failed. No retries permitted until 2026-09-18 20:32:20.14074438 +0000 UTC m=+1859.191104805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls") pod "maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" (UID: "aeaa3d9c-d121-469c-9a55-3e57392a1855") : secret "maas-api-e2e-ipp-a-8cdaea-serving-cert" not found Sep 18 20:32:19.641306 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.641270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa3d9c-d121-469c-9a55-3e57392a1855-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.643153 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.643133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-metrics-certs\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:19.651102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:19.651079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fmz\" (UniqueName: \"kubernetes.io/projected/aeaa3d9c-d121-469c-9a55-3e57392a1855-kube-api-access-w8fmz\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:20.145689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.145655 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:20.148183 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.148155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:20.154087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.154062 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s"] Sep 18 20:32:20.157901 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.157883 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.160472 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.160453 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8cdaea-dockercfg-jhltj\"" Sep 18 20:32:20.160580 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.160457 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8cdaea\"" Sep 18 20:32:20.167043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.167020 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s"] Sep 18 20:32:20.246240 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.246205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb81ed94-3f6f-4698-bca6-757a4f286d50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.246415 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.246360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnrv\" (UniqueName: \"kubernetes.io/projected/fb81ed94-3f6f-4698-bca6-757a4f286d50-kube-api-access-xcnrv\") pod \"payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.347661 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.347621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb81ed94-3f6f-4698-bca6-757a4f286d50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.347841 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.347695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnrv\" (UniqueName: \"kubernetes.io/projected/fb81ed94-3f6f-4698-bca6-757a4f286d50-kube-api-access-xcnrv\") pod \"payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.348354 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.348334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb81ed94-3f6f-4698-bca6-757a4f286d50-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.355859 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.355835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnrv\" (UniqueName: \"kubernetes.io/projected/fb81ed94-3f6f-4698-bca6-757a4f286d50-kube-api-access-xcnrv\") pod \"payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.413311 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.413193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:20.469612 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.469581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:20.687184 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.687099 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz"] Sep 18 20:32:20.690205 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:32:20.690170 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaa3d9c_d121_469c_9a55_3e57392a1855.slice/crio-19ff6aeb562b480f66aed5cd9d9fbe6a9ccd28fe48619b77b585a015efb73200 WatchSource:0}: Error finding container 19ff6aeb562b480f66aed5cd9d9fbe6a9ccd28fe48619b77b585a015efb73200: Status 404 returned error can't find the container with id 19ff6aeb562b480f66aed5cd9d9fbe6a9ccd28fe48619b77b585a015efb73200 Sep 18 20:32:20.755676 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:20.755649 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s"] Sep 18 20:32:20.757941 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:32:20.757917 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb81ed94_3f6f_4698_bca6_757a4f286d50.slice/crio-738f54744fd53b96ee0c31b27cad20dc93a4910e66a06dfa4c936d51e761645f WatchSource:0}: Error finding container 738f54744fd53b96ee0c31b27cad20dc93a4910e66a06dfa4c936d51e761645f: Status 404 returned error can't find the container with id 738f54744fd53b96ee0c31b27cad20dc93a4910e66a06dfa4c936d51e761645f Sep 18 20:32:21.445519 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.445486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" event={"ID":"aeaa3d9c-d121-469c-9a55-3e57392a1855","Type":"ContainerStarted","Data":"15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac"} Sep 18 20:32:21.445769 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.445542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" event={"ID":"aeaa3d9c-d121-469c-9a55-3e57392a1855","Type":"ContainerStarted","Data":"19ff6aeb562b480f66aed5cd9d9fbe6a9ccd28fe48619b77b585a015efb73200"} Sep 18 20:32:21.445769 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.445575 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:21.447157 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.447129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" event={"ID":"fb81ed94-3f6f-4698-bca6-757a4f286d50","Type":"ContainerStarted","Data":"a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec"} Sep 18 20:32:21.447285 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.447180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" event={"ID":"fb81ed94-3f6f-4698-bca6-757a4f286d50","Type":"ContainerStarted","Data":"738f54744fd53b96ee0c31b27cad20dc93a4910e66a06dfa4c936d51e761645f"} Sep 18 20:32:21.447285 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.447218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:21.463898 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.463851 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" podStartSLOduration=2.14576507 podStartE2EDuration="2.463835174s" totalImagesPullingTime="318.070104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:21.461901288 +0000 UTC m=+1860.512261738" watchObservedRunningTime="2026-09-18 20:32:21.463835174 +0000 UTC m=+1860.514195621" Sep 18 20:32:21.478429 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:21.478392 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" podStartSLOduration=1.478377048 podStartE2EDuration="1.478377048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:21.477746995 +0000 UTC m=+1860.528107442" watchObservedRunningTime="2026-09-18 20:32:21.478377048 +0000 UTC m=+1860.528737508" Sep 18 20:32:27.458520 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:27.458487 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:32:29.119689 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.119626 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" podUID="9a89ddbc-80da-423a-90e3-c73dcf03b062" containerName="main" containerID="cri-o://393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35" gracePeriod=45 Sep 18 20:32:29.368160 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.368136 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:32:29.432743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432660 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-tmp-dir\") pod \"9a89ddbc-80da-423a-90e3-c73dcf03b062\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " Sep 18 20:32:29.432743 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a89ddbc-80da-423a-90e3-c73dcf03b062-tls-certs\") pod \"9a89ddbc-80da-423a-90e3-c73dcf03b062\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " Sep 18 20:32:29.432934 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432751 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x57wf\" (UniqueName: \"kubernetes.io/projected/9a89ddbc-80da-423a-90e3-c73dcf03b062-kube-api-access-x57wf\") pod \"9a89ddbc-80da-423a-90e3-c73dcf03b062\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " Sep 18 20:32:29.432934 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-model-cache\") pod \"9a89ddbc-80da-423a-90e3-c73dcf03b062\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " Sep 18 20:32:29.432934 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432842 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-dshm\") pod \"9a89ddbc-80da-423a-90e3-c73dcf03b062\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " Sep 18 20:32:29.432934 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432902 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-home\") pod \"9a89ddbc-80da-423a-90e3-c73dcf03b062\" (UID: \"9a89ddbc-80da-423a-90e3-c73dcf03b062\") " Sep 18 20:32:29.433146 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.432929 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a89ddbc-80da-423a-90e3-c73dcf03b062" (UID: "9a89ddbc-80da-423a-90e3-c73dcf03b062"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:32:29.433146 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.433101 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a89ddbc-80da-423a-90e3-c73dcf03b062" (UID: "9a89ddbc-80da-423a-90e3-c73dcf03b062"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:32:29.433260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.433155 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-home" (OuterVolumeSpecName: "home") pod "9a89ddbc-80da-423a-90e3-c73dcf03b062" (UID: "9a89ddbc-80da-423a-90e3-c73dcf03b062"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:32:29.433260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.433212 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-tmp-dir\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:32:29.433260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.433230 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-model-cache\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:32:29.434976 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.434947 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a89ddbc-80da-423a-90e3-c73dcf03b062-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a89ddbc-80da-423a-90e3-c73dcf03b062" (UID: "9a89ddbc-80da-423a-90e3-c73dcf03b062"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:32:29.434976 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.434969 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-dshm" (OuterVolumeSpecName: "dshm") pod "9a89ddbc-80da-423a-90e3-c73dcf03b062" (UID: "9a89ddbc-80da-423a-90e3-c73dcf03b062"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 20:32:29.435122 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.434992 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a89ddbc-80da-423a-90e3-c73dcf03b062-kube-api-access-x57wf" (OuterVolumeSpecName: "kube-api-access-x57wf") pod "9a89ddbc-80da-423a-90e3-c73dcf03b062" (UID: "9a89ddbc-80da-423a-90e3-c73dcf03b062"). InnerVolumeSpecName "kube-api-access-x57wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:32:29.497676 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.497638 2559 generic.go:356] "Generic (PLEG): container finished" podID="9a89ddbc-80da-423a-90e3-c73dcf03b062" containerID="393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35" exitCode=0 Sep 18 20:32:29.497866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.497686 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" Sep 18 20:32:29.497866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.497707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" event={"ID":"9a89ddbc-80da-423a-90e3-c73dcf03b062","Type":"ContainerDied","Data":"393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35"} Sep 18 20:32:29.497866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.497754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv" event={"ID":"9a89ddbc-80da-423a-90e3-c73dcf03b062","Type":"ContainerDied","Data":"677ec0ca5e28a8c9db38dd28a7adb806dbf116d71d5c2f2935f04e7fccbf876d"} Sep 18 20:32:29.497866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.497774 2559 scope.go:117] "RemoveContainer" containerID="393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35" Sep 18 20:32:29.506751 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.506732 2559 scope.go:117] "RemoveContainer" containerID="393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35" Sep 18 20:32:29.507015 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:32:29.506996 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35\": container with ID starting with 393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35 not found: ID does not exist" containerID="393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35" Sep 18 20:32:29.507087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.507023 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35"} err="failed to get container status \"393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35\": rpc error: code = NotFound desc = could not find container \"393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35\": container with ID starting with 393293b6e511615a3d8841e5d360bfef6424cc641af0109fb236bcbc32738e35 not found: ID does not exist" Sep 18 20:32:29.519778 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.519752 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv"] Sep 18 20:32:29.523483 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.523462 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-35558b/rate-test-model-35558b-kserve-54fbc87cdc-ncfnv"] Sep 18 20:32:29.533988 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.533969 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-home\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:32:29.534048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.533991 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a89ddbc-80da-423a-90e3-c73dcf03b062-tls-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:32:29.534048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.534000 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57wf\" (UniqueName: \"kubernetes.io/projected/9a89ddbc-80da-423a-90e3-c73dcf03b062-kube-api-access-x57wf\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:32:29.534048 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:29.534009 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a89ddbc-80da-423a-90e3-c73dcf03b062-dshm\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:32:31.184502 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:31.184469 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a89ddbc-80da-423a-90e3-c73dcf03b062" path="/var/lib/kubelet/pods/9a89ddbc-80da-423a-90e3-c73dcf03b062/volumes" Sep 18 20:32:32.454099 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:32.454067 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:32:49.481340 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.481233 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd"] Sep 18 20:32:49.481966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.481836 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a89ddbc-80da-423a-90e3-c73dcf03b062" containerName="main" Sep 18 20:32:49.481966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.481853 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a89ddbc-80da-423a-90e3-c73dcf03b062" containerName="main" Sep 18 20:32:49.481966 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.481945 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a89ddbc-80da-423a-90e3-c73dcf03b062" containerName="main" Sep 18 20:32:49.485379 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.485347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.488115 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.488093 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-667e4c-serving-cert\"" Sep 18 20:32:49.497421 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.497390 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd"] Sep 18 20:32:49.610831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.610792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-metrics-certs\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.611037 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.610857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6q24\" (UniqueName: \"kubernetes.io/projected/3b1223d6-f0d2-4926-8ce8-d1bead24229d-kube-api-access-r6q24\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.611037 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.610895 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.611037 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.610916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1223d6-f0d2-4926-8ce8-d1bead24229d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.711789 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.711759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.711983 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.711799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1223d6-f0d2-4926-8ce8-d1bead24229d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.711983 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.711855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-metrics-certs\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.711983 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.711904 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6q24\" (UniqueName: \"kubernetes.io/projected/3b1223d6-f0d2-4926-8ce8-d1bead24229d-kube-api-access-r6q24\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.712622 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.712600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1223d6-f0d2-4926-8ce8-d1bead24229d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.714251 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.714227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.714363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.714346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-metrics-certs\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.719989 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.719969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6q24\" (UniqueName: \"kubernetes.io/projected/3b1223d6-f0d2-4926-8ce8-d1bead24229d-kube-api-access-r6q24\") pod \"maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:49.797866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:49.797797 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:50.083389 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:50.083363 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd"] Sep 18 20:32:50.086209 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:32:50.086183 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1223d6_f0d2_4926_8ce8_d1bead24229d.slice/crio-4f46c69d0ce99d57387d1ebdeb4d85d0e449880900f7735ea96d358396cef26c WatchSource:0}: Error finding container 4f46c69d0ce99d57387d1ebdeb4d85d0e449880900f7735ea96d358396cef26c: Status 404 returned error can't find the container with id 4f46c69d0ce99d57387d1ebdeb4d85d0e449880900f7735ea96d358396cef26c Sep 18 20:32:50.606633 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:50.606593 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" event={"ID":"3b1223d6-f0d2-4926-8ce8-d1bead24229d","Type":"ContainerStarted","Data":"093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787"} Sep 18 20:32:50.607094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:50.606656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" event={"ID":"3b1223d6-f0d2-4926-8ce8-d1bead24229d","Type":"ContainerStarted","Data":"4f46c69d0ce99d57387d1ebdeb4d85d0e449880900f7735ea96d358396cef26c"} Sep 18 20:32:50.607094 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:50.606691 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:32:50.625866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:50.625822 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" podStartSLOduration=1.280788554 podStartE2EDuration="1.625807501s" totalImagesPullingTime="345.018947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:50.62324698 +0000 UTC m=+1889.673607442" watchObservedRunningTime="2026-09-18 20:32:50.625807501 +0000 UTC m=+1889.676167949" Sep 18 20:32:56.617861 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:32:56.617832 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:33:17.288912 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:33:17.288882 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:33:17.397740 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:33:17.397701 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:33:53.420454 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:33:53.420421 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s_fb81ed94-3f6f-4698-bca6-757a4f286d50/payload-processing/0.log" Sep 18 20:33:53.531719 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:33:53.531694 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:33:53.647195 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:33:53.647170 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:34:00.195236 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.195202 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5"] Sep 18 20:34:00.198910 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.198887 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.201568 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.201547 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-186465-serving-cert\"" Sep 18 20:34:00.210754 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.210729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5"] Sep 18 20:34:00.254620 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.254588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878jq\" (UniqueName: \"kubernetes.io/projected/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-kube-api-access-878jq\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.254757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.254624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.254757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.254678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.254843 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.254807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.356077 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.356045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878jq\" (UniqueName: \"kubernetes.io/projected/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-kube-api-access-878jq\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.356077 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.356081 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.356338 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.356107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.356338 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.356169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.356338 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:00.356259 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-serving-cert: secret "maas-api-e2e-ipp-cleanup-186465-serving-cert" not found Sep 18 20:34:00.356338 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:00.356330 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls podName:d776d4c8-b1c2-4b8a-9188-eb63796cdce1 nodeName:}" failed. No retries permitted until 2026-09-18 20:34:00.856315042 +0000 UTC m=+1959.906675468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" (UID: "d776d4c8-b1c2-4b8a-9188-eb63796cdce1") : secret "maas-api-e2e-ipp-cleanup-186465-serving-cert" not found Sep 18 20:34:00.356818 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.356794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.358662 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.358634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.365895 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.365871 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-878jq\" (UniqueName: \"kubernetes.io/projected/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-kube-api-access-878jq\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.762650 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.762613 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25"] Sep 18 20:34:00.766392 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.766372 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.769014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.768993 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-186465\"" Sep 18 20:34:00.769159 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.769140 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-186465-dockercfg-g9rjc\"" Sep 18 20:34:00.777126 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.777103 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25"] Sep 18 20:34:00.818931 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.818910 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m"] Sep 18 20:34:00.822306 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.822265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:00.832647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.832623 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m"] Sep 18 20:34:00.860605 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.860570 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e957f88c-b110-42d2-8425-64145a2bfd4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.860605 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.860617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbgv\" (UniqueName: \"kubernetes.io/projected/e957f88c-b110-42d2-8425-64145a2bfd4c-kube-api-access-8qbgv\") pod \"payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.860877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.860701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.860877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.860745 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsc6q\" (UniqueName: \"kubernetes.io/projected/21e57d02-10f0-43d9-ae3e-a3025ae00431-kube-api-access-zsc6q\") pod \"payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:00.860877 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.860768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e57d02-10f0-43d9-ae3e-a3025ae00431-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:00.866243 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.866211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:00.961679 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.961618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e957f88c-b110-42d2-8425-64145a2bfd4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.961679 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.961687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbgv\" (UniqueName: \"kubernetes.io/projected/e957f88c-b110-42d2-8425-64145a2bfd4c-kube-api-access-8qbgv\") pod \"payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.961905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.961761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsc6q\" (UniqueName: \"kubernetes.io/projected/21e57d02-10f0-43d9-ae3e-a3025ae00431-kube-api-access-zsc6q\") pod \"payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:00.961905 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.961785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e57d02-10f0-43d9-ae3e-a3025ae00431-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:00.962382 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.962362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e957f88c-b110-42d2-8425-64145a2bfd4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.962438 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.962422 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e57d02-10f0-43d9-ae3e-a3025ae00431-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:00.970393 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.970368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbgv\" (UniqueName: \"kubernetes.io/projected/e957f88c-b110-42d2-8425-64145a2bfd4c-kube-api-access-8qbgv\") pod \"payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:00.970513 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:00.970374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsc6q\" (UniqueName: \"kubernetes.io/projected/21e57d02-10f0-43d9-ae3e-a3025ae00431-kube-api-access-zsc6q\") pod \"payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:01.078207 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.078112 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:01.112228 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.112189 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:01.133589 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.133528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:01.352198 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.352168 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25"] Sep 18 20:34:01.353629 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:34:01.353606 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode957f88c_b110_42d2_8425_64145a2bfd4c.slice/crio-c3a3cdd81a1a444867ff4a99d7d142796fda8ee1b792d22a47b2e11fa60a531e WatchSource:0}: Error finding container c3a3cdd81a1a444867ff4a99d7d142796fda8ee1b792d22a47b2e11fa60a531e: Status 404 returned error can't find the container with id c3a3cdd81a1a444867ff4a99d7d142796fda8ee1b792d22a47b2e11fa60a531e Sep 18 20:34:01.403567 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.403543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5"] Sep 18 20:34:01.405682 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:34:01.405647 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd776d4c8_b1c2_4b8a_9188_eb63796cdce1.slice/crio-215bd86ac0098943002050f24bbf87ea16ffc6c1b02e4e8abb3492b8446c089c WatchSource:0}: Error finding container 215bd86ac0098943002050f24bbf87ea16ffc6c1b02e4e8abb3492b8446c089c: Status 404 returned error can't find the container with id 215bd86ac0098943002050f24bbf87ea16ffc6c1b02e4e8abb3492b8446c089c Sep 18 20:34:01.407872 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.407847 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:34:01.429760 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.424495 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m"] Sep 18 20:34:01.433127 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:34:01.433098 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e57d02_10f0_43d9_ae3e_a3025ae00431.slice/crio-3f8a20127523c073555af7ab6e0e6f3d6f70c2ce99eadae02c49dcbfa72de250 WatchSource:0}: Error finding container 3f8a20127523c073555af7ab6e0e6f3d6f70c2ce99eadae02c49dcbfa72de250: Status 404 returned error can't find the container with id 3f8a20127523c073555af7ab6e0e6f3d6f70c2ce99eadae02c49dcbfa72de250 Sep 18 20:34:01.988570 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.988531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" event={"ID":"e957f88c-b110-42d2-8425-64145a2bfd4c","Type":"ContainerStarted","Data":"e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f"} Sep 18 20:34:01.988755 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.988593 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" event={"ID":"e957f88c-b110-42d2-8425-64145a2bfd4c","Type":"ContainerStarted","Data":"c3a3cdd81a1a444867ff4a99d7d142796fda8ee1b792d22a47b2e11fa60a531e"} Sep 18 20:34:01.988755 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.988625 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:01.990133 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.990107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" event={"ID":"d776d4c8-b1c2-4b8a-9188-eb63796cdce1","Type":"ContainerStarted","Data":"8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467"} Sep 18 20:34:01.990250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.990148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" event={"ID":"d776d4c8-b1c2-4b8a-9188-eb63796cdce1","Type":"ContainerStarted","Data":"215bd86ac0098943002050f24bbf87ea16ffc6c1b02e4e8abb3492b8446c089c"} Sep 18 20:34:01.990250 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.990210 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:01.991767 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.991744 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" event={"ID":"21e57d02-10f0-43d9-ae3e-a3025ae00431","Type":"ContainerStarted","Data":"3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b"} Sep 18 20:34:01.991860 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.991790 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" event={"ID":"21e57d02-10f0-43d9-ae3e-a3025ae00431","Type":"ContainerStarted","Data":"3f8a20127523c073555af7ab6e0e6f3d6f70c2ce99eadae02c49dcbfa72de250"} Sep 18 20:34:01.991922 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:01.991892 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:02.005329 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:02.005258 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" podStartSLOduration=2.005242322 podStartE2EDuration="2.005242322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:02.004163901 +0000 UTC m=+1961.054524363" watchObservedRunningTime="2026-09-18 20:34:02.005242322 +0000 UTC m=+1961.055602770" Sep 18 20:34:02.021202 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:02.021150 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" podStartSLOduration=1.546294154 podStartE2EDuration="2.021133646s" totalImagesPullingTime="474.839492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:02.021003977 +0000 UTC m=+1961.071364437" watchObservedRunningTime="2026-09-18 20:34:02.021133646 +0000 UTC m=+1961.071494125" Sep 18 20:34:02.038319 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:02.038255 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" podStartSLOduration=2.038239133 podStartE2EDuration="2.038239133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:02.036960385 +0000 UTC m=+1961.087320836" watchObservedRunningTime="2026-09-18 20:34:02.038239133 +0000 UTC m=+1961.088599616" Sep 18 20:34:08.003678 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:08.003598 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:13.000836 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:13.000804 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:13.001256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:13.000883 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:29.187384 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.187340 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5"] Sep 18 20:34:29.187876 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.187575 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" podUID="d776d4c8-b1c2-4b8a-9188-eb63796cdce1" containerName="maas-api" containerID="cri-o://8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467" gracePeriod=30 Sep 18 20:34:29.249680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.249397 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m"] Sep 18 20:34:29.249680 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.249647 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" podUID="21e57d02-10f0-43d9-ae3e-a3025ae00431" containerName="payload-processing" containerID="cri-o://3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b" gracePeriod=30 Sep 18 20:34:29.252710 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.252681 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25"] Sep 18 20:34:29.253054 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.252999 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" podUID="e957f88c-b110-42d2-8425-64145a2bfd4c" containerName="payload-pre-processing" containerID="cri-o://e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f" gracePeriod=30 Sep 18 20:34:29.521780 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.521748 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:29.527727 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.527699 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-878jq\" (UniqueName: \"kubernetes.io/projected/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-kube-api-access-878jq\") pod \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " Sep 18 20:34:29.527889 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.527741 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-metrics-certs\") pod \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " Sep 18 20:34:29.527889 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.527765 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls\") pod \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " Sep 18 20:34:29.527889 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.527816 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-openshift-service-ca\") pod \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\" (UID: \"d776d4c8-b1c2-4b8a-9188-eb63796cdce1\") " Sep 18 20:34:29.528443 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.528412 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d776d4c8-b1c2-4b8a-9188-eb63796cdce1" (UID: "d776d4c8-b1c2-4b8a-9188-eb63796cdce1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:34:29.531982 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.531917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d776d4c8-b1c2-4b8a-9188-eb63796cdce1" (UID: "d776d4c8-b1c2-4b8a-9188-eb63796cdce1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:34:29.531982 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.531949 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-kube-api-access-878jq" (OuterVolumeSpecName: "kube-api-access-878jq") pod "d776d4c8-b1c2-4b8a-9188-eb63796cdce1" (UID: "d776d4c8-b1c2-4b8a-9188-eb63796cdce1"). InnerVolumeSpecName "kube-api-access-878jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:34:29.532116 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.532014 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d776d4c8-b1c2-4b8a-9188-eb63796cdce1" (UID: "d776d4c8-b1c2-4b8a-9188-eb63796cdce1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:34:29.562096 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.562063 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:29.572020 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.571996 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:29.629149 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629118 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e57d02-10f0-43d9-ae3e-a3025ae00431-plugins-config-volume\") pod \"21e57d02-10f0-43d9-ae3e-a3025ae00431\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " Sep 18 20:34:29.629371 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629165 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qbgv\" (UniqueName: \"kubernetes.io/projected/e957f88c-b110-42d2-8425-64145a2bfd4c-kube-api-access-8qbgv\") pod \"e957f88c-b110-42d2-8425-64145a2bfd4c\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " Sep 18 20:34:29.629371 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629206 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e957f88c-b110-42d2-8425-64145a2bfd4c-plugins-config-volume\") pod \"e957f88c-b110-42d2-8425-64145a2bfd4c\" (UID: \"e957f88c-b110-42d2-8425-64145a2bfd4c\") " Sep 18 20:34:29.629371 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsc6q\" (UniqueName: \"kubernetes.io/projected/21e57d02-10f0-43d9-ae3e-a3025ae00431-kube-api-access-zsc6q\") pod \"21e57d02-10f0-43d9-ae3e-a3025ae00431\" (UID: \"21e57d02-10f0-43d9-ae3e-a3025ae00431\") " Sep 18 20:34:29.629555 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629535 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.629614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629561 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878jq\" (UniqueName: \"kubernetes.io/projected/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-kube-api-access-878jq\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.629614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629575 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.629614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629564 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e57d02-10f0-43d9-ae3e-a3025ae00431-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21e57d02-10f0-43d9-ae3e-a3025ae00431" (UID: "21e57d02-10f0-43d9-ae3e-a3025ae00431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:34:29.629614 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629589 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d776d4c8-b1c2-4b8a-9188-eb63796cdce1-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.629814 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.629635 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e957f88c-b110-42d2-8425-64145a2bfd4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e957f88c-b110-42d2-8425-64145a2bfd4c" (UID: "e957f88c-b110-42d2-8425-64145a2bfd4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:34:29.631549 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.631521 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e957f88c-b110-42d2-8425-64145a2bfd4c-kube-api-access-8qbgv" (OuterVolumeSpecName: "kube-api-access-8qbgv") pod "e957f88c-b110-42d2-8425-64145a2bfd4c" (UID: "e957f88c-b110-42d2-8425-64145a2bfd4c"). InnerVolumeSpecName "kube-api-access-8qbgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:34:29.631652 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.631566 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e57d02-10f0-43d9-ae3e-a3025ae00431-kube-api-access-zsc6q" (OuterVolumeSpecName: "kube-api-access-zsc6q") pod "21e57d02-10f0-43d9-ae3e-a3025ae00431" (UID: "21e57d02-10f0-43d9-ae3e-a3025ae00431"). InnerVolumeSpecName "kube-api-access-zsc6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:34:29.730575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.730539 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e57d02-10f0-43d9-ae3e-a3025ae00431-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.730575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.730564 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbgv\" (UniqueName: \"kubernetes.io/projected/e957f88c-b110-42d2-8425-64145a2bfd4c-kube-api-access-8qbgv\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.730575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.730576 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e957f88c-b110-42d2-8425-64145a2bfd4c-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:29.730575 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:29.730586 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsc6q\" (UniqueName: \"kubernetes.io/projected/21e57d02-10f0-43d9-ae3e-a3025ae00431-kube-api-access-zsc6q\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:30.142105 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.142069 2559 generic.go:356] "Generic (PLEG): container finished" podID="d776d4c8-b1c2-4b8a-9188-eb63796cdce1" containerID="8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467" exitCode=0 Sep 18 20:34:30.142331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.142127 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" Sep 18 20:34:30.142331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.142157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" event={"ID":"d776d4c8-b1c2-4b8a-9188-eb63796cdce1","Type":"ContainerDied","Data":"8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467"} Sep 18 20:34:30.142331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.142235 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5" event={"ID":"d776d4c8-b1c2-4b8a-9188-eb63796cdce1","Type":"ContainerDied","Data":"215bd86ac0098943002050f24bbf87ea16ffc6c1b02e4e8abb3492b8446c089c"} Sep 18 20:34:30.142331 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.142253 2559 scope.go:117] "RemoveContainer" containerID="8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467" Sep 18 20:34:30.143631 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.143611 2559 generic.go:356] "Generic (PLEG): container finished" podID="21e57d02-10f0-43d9-ae3e-a3025ae00431" containerID="3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b" exitCode=0 Sep 18 20:34:30.143757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.143671 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" Sep 18 20:34:30.143757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.143687 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" event={"ID":"21e57d02-10f0-43d9-ae3e-a3025ae00431","Type":"ContainerDied","Data":"3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b"} Sep 18 20:34:30.143757 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.143756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m" event={"ID":"21e57d02-10f0-43d9-ae3e-a3025ae00431","Type":"ContainerDied","Data":"3f8a20127523c073555af7ab6e0e6f3d6f70c2ce99eadae02c49dcbfa72de250"} Sep 18 20:34:30.145456 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.145142 2559 generic.go:356] "Generic (PLEG): container finished" podID="e957f88c-b110-42d2-8425-64145a2bfd4c" containerID="e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f" exitCode=0 Sep 18 20:34:30.145456 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.145200 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" event={"ID":"e957f88c-b110-42d2-8425-64145a2bfd4c","Type":"ContainerDied","Data":"e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f"} Sep 18 20:34:30.145456 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.145240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" event={"ID":"e957f88c-b110-42d2-8425-64145a2bfd4c","Type":"ContainerDied","Data":"c3a3cdd81a1a444867ff4a99d7d142796fda8ee1b792d22a47b2e11fa60a531e"} Sep 18 20:34:30.145456 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.145324 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25" Sep 18 20:34:30.152974 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.152954 2559 scope.go:117] "RemoveContainer" containerID="8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467" Sep 18 20:34:30.153577 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:30.153408 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467\": container with ID starting with 8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467 not found: ID does not exist" containerID="8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467" Sep 18 20:34:30.153577 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.153446 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467"} err="failed to get container status \"8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467\": rpc error: code = NotFound desc = could not find container \"8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467\": container with ID starting with 8e86d2c75c761c6a920c7fbb0265b0bc4fe38a9db04aad2f5ddccbc68ed82467 not found: ID does not exist" Sep 18 20:34:30.153577 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.153469 2559 scope.go:117] "RemoveContainer" containerID="3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b" Sep 18 20:34:30.162732 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.162704 2559 scope.go:117] "RemoveContainer" containerID="3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b" Sep 18 20:34:30.162988 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:30.162970 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b\": container with ID starting with 3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b not found: ID does not exist" containerID="3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b" Sep 18 20:34:30.163043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.162993 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b"} err="failed to get container status \"3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b\": rpc error: code = NotFound desc = could not find container \"3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b\": container with ID starting with 3bb21e3d1336a3dc0495ab0b7dbe3c010a1c6c26227831bc686992bc6cf87e6b not found: ID does not exist" Sep 18 20:34:30.163043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.163009 2559 scope.go:117] "RemoveContainer" containerID="e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f" Sep 18 20:34:30.171792 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.171778 2559 scope.go:117] "RemoveContainer" containerID="e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f" Sep 18 20:34:30.172067 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:30.172051 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f\": container with ID starting with e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f not found: ID does not exist" containerID="e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f" Sep 18 20:34:30.172118 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.172072 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f"} err="failed to get container status \"e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f\": rpc error: code = NotFound desc = could not find container \"e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f\": container with ID starting with e6e64b36ae1d963322c2bac1b48bf243fffad652d154ccf43eefb5c3f724247f not found: ID does not exist" Sep 18 20:34:30.172499 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.172480 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5"] Sep 18 20:34:30.175386 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.175366 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-186465-7fc86b9b96-v8lz5"] Sep 18 20:34:30.184644 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.184614 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m"] Sep 18 20:34:30.188681 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.188659 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-186465-d86bc4d8c-hl98m"] Sep 18 20:34:30.197657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.197637 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25"] Sep 18 20:34:30.203352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:30.203249 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-186465-64788c88c4-4bv25"] Sep 18 20:34:31.183362 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:31.183327 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e57d02-10f0-43d9-ae3e-a3025ae00431" path="/var/lib/kubelet/pods/21e57d02-10f0-43d9-ae3e-a3025ae00431/volumes" Sep 18 20:34:31.183739 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:31.183725 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d776d4c8-b1c2-4b8a-9188-eb63796cdce1" path="/var/lib/kubelet/pods/d776d4c8-b1c2-4b8a-9188-eb63796cdce1/volumes" Sep 18 20:34:31.184043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:31.184031 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e957f88c-b110-42d2-8425-64145a2bfd4c" path="/var/lib/kubelet/pods/e957f88c-b110-42d2-8425-64145a2bfd4c/volumes" Sep 18 20:34:54.690847 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:54.690804 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz"] Sep 18 20:34:54.691270 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:54.691074 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" podUID="aeaa3d9c-d121-469c-9a55-3e57392a1855" containerName="maas-api" containerID="cri-o://15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac" gracePeriod=30 Sep 18 20:34:54.753574 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:54.753537 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s"] Sep 18 20:34:54.754014 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:54.753955 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" podUID="fb81ed94-3f6f-4698-bca6-757a4f286d50" containerName="payload-processing" containerID="cri-o://a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec" gracePeriod=30 Sep 18 20:34:55.008668 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.008637 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:34:55.048098 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.048064 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:34:55.151263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.151228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcnrv\" (UniqueName: \"kubernetes.io/projected/fb81ed94-3f6f-4698-bca6-757a4f286d50-kube-api-access-xcnrv\") pod \"fb81ed94-3f6f-4698-bca6-757a4f286d50\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " Sep 18 20:34:55.151457 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.151328 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa3d9c-d121-469c-9a55-3e57392a1855-openshift-service-ca\") pod \"aeaa3d9c-d121-469c-9a55-3e57392a1855\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " Sep 18 20:34:55.151882 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.151575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls\") pod \"aeaa3d9c-d121-469c-9a55-3e57392a1855\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " Sep 18 20:34:55.151882 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.151657 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8fmz\" (UniqueName: \"kubernetes.io/projected/aeaa3d9c-d121-469c-9a55-3e57392a1855-kube-api-access-w8fmz\") pod \"aeaa3d9c-d121-469c-9a55-3e57392a1855\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " Sep 18 20:34:55.151882 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.151709 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeaa3d9c-d121-469c-9a55-3e57392a1855-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aeaa3d9c-d121-469c-9a55-3e57392a1855" (UID: "aeaa3d9c-d121-469c-9a55-3e57392a1855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:34:55.151882 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.151742 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb81ed94-3f6f-4698-bca6-757a4f286d50-plugins-config-volume\") pod \"fb81ed94-3f6f-4698-bca6-757a4f286d50\" (UID: \"fb81ed94-3f6f-4698-bca6-757a4f286d50\") " Sep 18 20:34:55.152766 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.152367 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-metrics-certs\") pod \"aeaa3d9c-d121-469c-9a55-3e57392a1855\" (UID: \"aeaa3d9c-d121-469c-9a55-3e57392a1855\") " Sep 18 20:34:55.152766 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.152362 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb81ed94-3f6f-4698-bca6-757a4f286d50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb81ed94-3f6f-4698-bca6-757a4f286d50" (UID: "fb81ed94-3f6f-4698-bca6-757a4f286d50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:34:55.152766 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.152722 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb81ed94-3f6f-4698-bca6-757a4f286d50-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:55.152766 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.152742 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa3d9c-d121-469c-9a55-3e57392a1855-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:55.153934 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.153900 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb81ed94-3f6f-4698-bca6-757a4f286d50-kube-api-access-xcnrv" (OuterVolumeSpecName: "kube-api-access-xcnrv") pod "fb81ed94-3f6f-4698-bca6-757a4f286d50" (UID: "fb81ed94-3f6f-4698-bca6-757a4f286d50"). InnerVolumeSpecName "kube-api-access-xcnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:34:55.154564 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.154533 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaa3d9c-d121-469c-9a55-3e57392a1855-kube-api-access-w8fmz" (OuterVolumeSpecName: "kube-api-access-w8fmz") pod "aeaa3d9c-d121-469c-9a55-3e57392a1855" (UID: "aeaa3d9c-d121-469c-9a55-3e57392a1855"). InnerVolumeSpecName "kube-api-access-w8fmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:34:55.155062 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.155026 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aeaa3d9c-d121-469c-9a55-3e57392a1855" (UID: "aeaa3d9c-d121-469c-9a55-3e57392a1855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:34:55.155295 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.155247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aeaa3d9c-d121-469c-9a55-3e57392a1855" (UID: "aeaa3d9c-d121-469c-9a55-3e57392a1855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:34:55.254352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.254227 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnrv\" (UniqueName: \"kubernetes.io/projected/fb81ed94-3f6f-4698-bca6-757a4f286d50-kube-api-access-xcnrv\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:55.254352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.254261 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:55.254352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.254287 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fmz\" (UniqueName: \"kubernetes.io/projected/aeaa3d9c-d121-469c-9a55-3e57392a1855-kube-api-access-w8fmz\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:55.254352 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.254301 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeaa3d9c-d121-469c-9a55-3e57392a1855-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:34:55.279630 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.279594 2559 generic.go:356] "Generic (PLEG): container finished" podID="fb81ed94-3f6f-4698-bca6-757a4f286d50" containerID="a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec" exitCode=0 Sep 18 20:34:55.279799 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.279641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" Sep 18 20:34:55.279799 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.279666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" event={"ID":"fb81ed94-3f6f-4698-bca6-757a4f286d50","Type":"ContainerDied","Data":"a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec"} Sep 18 20:34:55.279799 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.279726 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s" event={"ID":"fb81ed94-3f6f-4698-bca6-757a4f286d50","Type":"ContainerDied","Data":"738f54744fd53b96ee0c31b27cad20dc93a4910e66a06dfa4c936d51e761645f"} Sep 18 20:34:55.279799 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.279759 2559 scope.go:117] "RemoveContainer" containerID="a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec" Sep 18 20:34:55.281330 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.281306 2559 generic.go:356] "Generic (PLEG): container finished" podID="aeaa3d9c-d121-469c-9a55-3e57392a1855" containerID="15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac" exitCode=0 Sep 18 20:34:55.281445 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.281343 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" Sep 18 20:34:55.281445 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.281402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" event={"ID":"aeaa3d9c-d121-469c-9a55-3e57392a1855","Type":"ContainerDied","Data":"15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac"} Sep 18 20:34:55.281445 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.281440 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz" event={"ID":"aeaa3d9c-d121-469c-9a55-3e57392a1855","Type":"ContainerDied","Data":"19ff6aeb562b480f66aed5cd9d9fbe6a9ccd28fe48619b77b585a015efb73200"} Sep 18 20:34:55.288884 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.288865 2559 scope.go:117] "RemoveContainer" containerID="a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec" Sep 18 20:34:55.289185 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:55.289156 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec\": container with ID starting with a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec not found: ID does not exist" containerID="a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec" Sep 18 20:34:55.289299 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.289187 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec"} err="failed to get container status \"a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec\": rpc error: code = NotFound desc = could not find container \"a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec\": container with ID starting with a268e18e44045b2565191c3edfa907e0d85aaf252538b35328cbf13acc3e0fec not found: ID does not exist" Sep 18 20:34:55.289299 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.289203 2559 scope.go:117] "RemoveContainer" containerID="15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac" Sep 18 20:34:55.297647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.297620 2559 scope.go:117] "RemoveContainer" containerID="15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac" Sep 18 20:34:55.297912 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:34:55.297893 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac\": container with ID starting with 15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac not found: ID does not exist" containerID="15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac" Sep 18 20:34:55.297984 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.297922 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac"} err="failed to get container status \"15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac\": rpc error: code = NotFound desc = could not find container \"15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac\": container with ID starting with 15229f041980de739eb07d90ad4cbc2edb4b6e3c70251b12e108905ba85642ac not found: ID does not exist" Sep 18 20:34:55.298983 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.298965 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz"] Sep 18 20:34:55.302249 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.302229 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cdaea-66494b89f7-cknfz"] Sep 18 20:34:55.313086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.313066 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s"] Sep 18 20:34:55.316650 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:55.316628 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8cdaea-6995844685-bd48s"] Sep 18 20:34:57.184479 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:57.184446 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeaa3d9c-d121-469c-9a55-3e57392a1855" path="/var/lib/kubelet/pods/aeaa3d9c-d121-469c-9a55-3e57392a1855/volumes" Sep 18 20:34:57.184891 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:34:57.184866 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb81ed94-3f6f-4698-bca6-757a4f286d50" path="/var/lib/kubelet/pods/fb81ed94-3f6f-4698-bca6-757a4f286d50/volumes" Sep 18 20:35:52.053454 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.053233 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd"] Sep 18 20:35:52.054043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.053549 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" podUID="3b1223d6-f0d2-4926-8ce8-d1bead24229d" containerName="maas-api" containerID="cri-o://093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787" gracePeriod=30 Sep 18 20:35:52.301705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.301675 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:35:52.359003 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.358974 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-maas-api-tls\") pod \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " Sep 18 20:35:52.359003 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.359007 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6q24\" (UniqueName: \"kubernetes.io/projected/3b1223d6-f0d2-4926-8ce8-d1bead24229d-kube-api-access-r6q24\") pod \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " Sep 18 20:35:52.359212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.359036 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1223d6-f0d2-4926-8ce8-d1bead24229d-openshift-service-ca\") pod \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " Sep 18 20:35:52.359212 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.359094 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-metrics-certs\") pod \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\" (UID: \"3b1223d6-f0d2-4926-8ce8-d1bead24229d\") " Sep 18 20:35:52.359499 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.359473 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1223d6-f0d2-4926-8ce8-d1bead24229d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b1223d6-f0d2-4926-8ce8-d1bead24229d" (UID: "3b1223d6-f0d2-4926-8ce8-d1bead24229d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:35:52.361801 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.361775 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b1223d6-f0d2-4926-8ce8-d1bead24229d" (UID: "3b1223d6-f0d2-4926-8ce8-d1bead24229d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:35:52.362061 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.362015 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b1223d6-f0d2-4926-8ce8-d1bead24229d" (UID: "3b1223d6-f0d2-4926-8ce8-d1bead24229d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:35:52.362348 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.362316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1223d6-f0d2-4926-8ce8-d1bead24229d-kube-api-access-r6q24" (OuterVolumeSpecName: "kube-api-access-r6q24") pod "3b1223d6-f0d2-4926-8ce8-d1bead24229d" (UID: "3b1223d6-f0d2-4926-8ce8-d1bead24229d"). InnerVolumeSpecName "kube-api-access-r6q24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:35:52.460952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.460920 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:35:52.460952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.460946 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6q24\" (UniqueName: \"kubernetes.io/projected/3b1223d6-f0d2-4926-8ce8-d1bead24229d-kube-api-access-r6q24\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:35:52.460952 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.460956 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1223d6-f0d2-4926-8ce8-d1bead24229d-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:35:52.461187 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.460965 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1223d6-f0d2-4926-8ce8-d1bead24229d-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:35:52.580366 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.580336 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b1223d6-f0d2-4926-8ce8-d1bead24229d" containerID="093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787" exitCode=0 Sep 18 20:35:52.580559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.580372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" event={"ID":"3b1223d6-f0d2-4926-8ce8-d1bead24229d","Type":"ContainerDied","Data":"093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787"} Sep 18 20:35:52.580559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.580391 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" Sep 18 20:35:52.580559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.580409 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd" event={"ID":"3b1223d6-f0d2-4926-8ce8-d1bead24229d","Type":"ContainerDied","Data":"4f46c69d0ce99d57387d1ebdeb4d85d0e449880900f7735ea96d358396cef26c"} Sep 18 20:35:52.580559 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.580436 2559 scope.go:117] "RemoveContainer" containerID="093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787" Sep 18 20:35:52.590104 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.590087 2559 scope.go:117] "RemoveContainer" containerID="093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787" Sep 18 20:35:52.590414 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:35:52.590389 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787\": container with ID starting with 093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787 not found: ID does not exist" containerID="093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787" Sep 18 20:35:52.590460 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.590422 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787"} err="failed to get container status \"093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787\": rpc error: code = NotFound desc = could not find container \"093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787\": container with ID starting with 093ed41ba1276c3b3bccd065b499a1ae46bdfebced2ac1493cdb7763961df787 not found: ID does not exist" Sep 18 20:35:52.601632 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.601608 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd"] Sep 18 20:35:52.604566 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:52.604546 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-667e4c-c98577cc9-c5vgd"] Sep 18 20:35:53.186327 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:35:53.186290 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1223d6-f0d2-4926-8ce8-d1bead24229d" path="/var/lib/kubelet/pods/3b1223d6-f0d2-4926-8ce8-d1bead24229d/volumes" Sep 18 20:36:09.759551 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:09.759518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn"] Sep 18 20:36:09.759992 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:09.759724 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" podUID="aacf64e7-5435-4590-a8c1-7f67bd61d9cc" containerName="payload-pre-processing" containerID="cri-o://465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060" gracePeriod=30 Sep 18 20:36:10.008054 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.008031 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:36:10.117458 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.117428 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f85df\" (UniqueName: \"kubernetes.io/projected/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-kube-api-access-f85df\") pod \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " Sep 18 20:36:10.117634 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.117488 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-plugins-config-volume\") pod \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\" (UID: \"aacf64e7-5435-4590-a8c1-7f67bd61d9cc\") " Sep 18 20:36:10.117850 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.117826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aacf64e7-5435-4590-a8c1-7f67bd61d9cc" (UID: "aacf64e7-5435-4590-a8c1-7f67bd61d9cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:36:10.119647 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.119621 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-kube-api-access-f85df" (OuterVolumeSpecName: "kube-api-access-f85df") pod "aacf64e7-5435-4590-a8c1-7f67bd61d9cc" (UID: "aacf64e7-5435-4590-a8c1-7f67bd61d9cc"). InnerVolumeSpecName "kube-api-access-f85df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:36:10.218988 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.218959 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85df\" (UniqueName: \"kubernetes.io/projected/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-kube-api-access-f85df\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:10.218988 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.218985 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf64e7-5435-4590-a8c1-7f67bd61d9cc-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:10.676912 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.676883 2559 generic.go:356] "Generic (PLEG): container finished" podID="aacf64e7-5435-4590-a8c1-7f67bd61d9cc" containerID="465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060" exitCode=0 Sep 18 20:36:10.677086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.676928 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" Sep 18 20:36:10.677086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.676976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" event={"ID":"aacf64e7-5435-4590-a8c1-7f67bd61d9cc","Type":"ContainerDied","Data":"465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060"} Sep 18 20:36:10.677086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.677024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn" event={"ID":"aacf64e7-5435-4590-a8c1-7f67bd61d9cc","Type":"ContainerDied","Data":"0622e892db4e0217cea4d2206e79640a206941b94e51ae069bf43c3343529c78"} Sep 18 20:36:10.677086 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.677048 2559 scope.go:117] "RemoveContainer" containerID="465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060" Sep 18 20:36:10.687106 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.687088 2559 scope.go:117] "RemoveContainer" containerID="465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060" Sep 18 20:36:10.687456 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:36:10.687435 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060\": container with ID starting with 465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060 not found: ID does not exist" containerID="465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060" Sep 18 20:36:10.687538 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.687466 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060"} err="failed to get container status \"465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060\": rpc error: code = NotFound desc = could not find container \"465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060\": container with ID starting with 465b10ea440b9fdc04d0279f608992409bc31db433997fdc29cd9a8a9495a060 not found: ID does not exist" Sep 18 20:36:10.701630 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.701606 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn"] Sep 18 20:36:10.705714 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:10.705690 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bab50d-9867fbb94-tltwn"] Sep 18 20:36:11.184292 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:11.184247 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aacf64e7-5435-4590-a8c1-7f67bd61d9cc" path="/var/lib/kubelet/pods/aacf64e7-5435-4590-a8c1-7f67bd61d9cc/volumes" Sep 18 20:36:23.302508 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:23.302480 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:36:23.306023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:23.306001 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:36:23.509652 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:23.509626 2559 scope.go:117] "RemoveContainer" containerID="20afe2cb5b58bd1babb83358c742da4bf9317295b9381c66235edd6410b588c0" Sep 18 20:36:27.483875 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.483840 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7"] Sep 18 20:36:27.487568 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.487536 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-35558b-29829390-svsm7"] Sep 18 20:36:27.510969 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.510936 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz"] Sep 18 20:36:27.511216 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.511170 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" podUID="bfe34356-d124-4917-b63a-87eb926021aa" containerName="payload-processing" containerID="cri-o://6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5" gracePeriod=30 Sep 18 20:36:27.762263 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.762235 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:36:27.762510 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.762491 2559 generic.go:356] "Generic (PLEG): container finished" podID="bfe34356-d124-4917-b63a-87eb926021aa" containerID="6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5" exitCode=0 Sep 18 20:36:27.762560 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.762534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" event={"ID":"bfe34356-d124-4917-b63a-87eb926021aa","Type":"ContainerDied","Data":"6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5"} Sep 18 20:36:27.762597 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.762580 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" event={"ID":"bfe34356-d124-4917-b63a-87eb926021aa","Type":"ContainerDied","Data":"b6f1f1ff66bc3ae7ba9a1809b398efbbadf1de55a054973f2b8d71920eb9fbf8"} Sep 18 20:36:27.762631 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.762601 2559 scope.go:117] "RemoveContainer" containerID="6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5" Sep 18 20:36:27.771156 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.771139 2559 scope.go:117] "RemoveContainer" containerID="6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5" Sep 18 20:36:27.771445 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:36:27.771426 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5\": container with ID starting with 6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5 not found: ID does not exist" containerID="6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5" Sep 18 20:36:27.771504 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.771451 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5"} err="failed to get container status \"6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5\": rpc error: code = NotFound desc = could not find container \"6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5\": container with ID starting with 6b9599f79194ea19a1fadd45c28ea8aae57f3eef6a742240aa72de7b161ec6e5 not found: ID does not exist" Sep 18 20:36:27.863960 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.863932 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t997n\" (UniqueName: \"kubernetes.io/projected/bfe34356-d124-4917-b63a-87eb926021aa-kube-api-access-t997n\") pod \"bfe34356-d124-4917-b63a-87eb926021aa\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " Sep 18 20:36:27.864143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.863973 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe34356-d124-4917-b63a-87eb926021aa-plugins-config-volume\") pod \"bfe34356-d124-4917-b63a-87eb926021aa\" (UID: \"bfe34356-d124-4917-b63a-87eb926021aa\") " Sep 18 20:36:27.864375 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.864348 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe34356-d124-4917-b63a-87eb926021aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfe34356-d124-4917-b63a-87eb926021aa" (UID: "bfe34356-d124-4917-b63a-87eb926021aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:36:27.866027 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.866002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe34356-d124-4917-b63a-87eb926021aa-kube-api-access-t997n" (OuterVolumeSpecName: "kube-api-access-t997n") pod "bfe34356-d124-4917-b63a-87eb926021aa" (UID: "bfe34356-d124-4917-b63a-87eb926021aa"). InnerVolumeSpecName "kube-api-access-t997n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:36:27.964738 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.964702 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t997n\" (UniqueName: \"kubernetes.io/projected/bfe34356-d124-4917-b63a-87eb926021aa-kube-api-access-t997n\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:27.964738 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:27.964731 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe34356-d124-4917-b63a-87eb926021aa-plugins-config-volume\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:28.768381 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:28.768351 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz" Sep 18 20:36:28.790446 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:28.790421 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz"] Sep 18 20:36:28.796774 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:28.796749 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-35558b-665f578f4c-mrlgz"] Sep 18 20:36:29.184236 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:29.184204 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe34356-d124-4917-b63a-87eb926021aa" path="/var/lib/kubelet/pods/bfe34356-d124-4917-b63a-87eb926021aa/volumes" Sep 18 20:36:29.184539 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:29.184524 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a85102-6b8b-49bf-b1df-b6a62534ffa6" path="/var/lib/kubelet/pods/e7a85102-6b8b-49bf-b1df-b6a62534ffa6/volumes" Sep 18 20:36:36.448926 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.448882 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fb768dd77-gpl6w"] Sep 18 20:36:36.449485 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.449131 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fb768dd77-gpl6w" podUID="57af45fb-aa04-4521-8273-40271aa49356" containerName="manager" containerID="cri-o://72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f" gracePeriod=30 Sep 18 20:36:36.686669 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.686637 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:36:36.747030 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.746934 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-metrics-certs\") pod \"57af45fb-aa04-4521-8273-40271aa49356\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " Sep 18 20:36:36.747199 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.747028 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsfm8\" (UniqueName: \"kubernetes.io/projected/57af45fb-aa04-4521-8273-40271aa49356-kube-api-access-wsfm8\") pod \"57af45fb-aa04-4521-8273-40271aa49356\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " Sep 18 20:36:36.747199 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.747102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-webhook-cert\") pod \"57af45fb-aa04-4521-8273-40271aa49356\" (UID: \"57af45fb-aa04-4521-8273-40271aa49356\") " Sep 18 20:36:36.748956 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.748923 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57af45fb-aa04-4521-8273-40271aa49356" (UID: "57af45fb-aa04-4521-8273-40271aa49356"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:36:36.749131 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.749117 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af45fb-aa04-4521-8273-40271aa49356-kube-api-access-wsfm8" (OuterVolumeSpecName: "kube-api-access-wsfm8") pod "57af45fb-aa04-4521-8273-40271aa49356" (UID: "57af45fb-aa04-4521-8273-40271aa49356"). InnerVolumeSpecName "kube-api-access-wsfm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:36:36.749416 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.749391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "57af45fb-aa04-4521-8273-40271aa49356" (UID: "57af45fb-aa04-4521-8273-40271aa49356"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:36:36.809466 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.809435 2559 generic.go:356] "Generic (PLEG): container finished" podID="57af45fb-aa04-4521-8273-40271aa49356" containerID="72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f" exitCode=0 Sep 18 20:36:36.809657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.809485 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-gpl6w" Sep 18 20:36:36.809657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.809512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-gpl6w" event={"ID":"57af45fb-aa04-4521-8273-40271aa49356","Type":"ContainerDied","Data":"72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f"} Sep 18 20:36:36.809657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.809562 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-gpl6w" event={"ID":"57af45fb-aa04-4521-8273-40271aa49356","Type":"ContainerDied","Data":"cf48a6bb75504a81013c337b3929def0724a3118ba75181f2eb7605622dccf5f"} Sep 18 20:36:36.809657 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.809584 2559 scope.go:117] "RemoveContainer" containerID="72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f" Sep 18 20:36:36.820299 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.820267 2559 scope.go:117] "RemoveContainer" containerID="72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f" Sep 18 20:36:36.820588 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:36:36.820571 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f\": container with ID starting with 72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f not found: ID does not exist" containerID="72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f" Sep 18 20:36:36.820641 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.820598 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f"} err="failed to get container status \"72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f\": rpc error: code = NotFound desc = could not find container \"72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f\": container with ID starting with 72df9c3a2b5e7a23410a0ebd69edb6fc21c204180672787957423073b99f481f not found: ID does not exist" Sep 18 20:36:36.831987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.831961 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fb768dd77-gpl6w"] Sep 18 20:36:36.835658 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.835634 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fb768dd77-gpl6w"] Sep 18 20:36:36.848681 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.848660 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsfm8\" (UniqueName: \"kubernetes.io/projected/57af45fb-aa04-4521-8273-40271aa49356-kube-api-access-wsfm8\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:36.848681 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.848681 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-webhook-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:36.848818 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:36.848691 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57af45fb-aa04-4521-8273-40271aa49356-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:36:37.183526 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:36:37.183496 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57af45fb-aa04-4521-8273-40271aa49356" path="/var/lib/kubelet/pods/57af45fb-aa04-4521-8273-40271aa49356/volumes" Sep 18 20:37:23.534177 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:23.534143 2559 scope.go:117] "RemoveContainer" containerID="6a6ef16637193d61d9b42d17de6fea40fdfe47a8c8348211e0de55d5733257d4" Sep 18 20:37:37.607471 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607435 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fb768dd77-tdpgp"] Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607775 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeaa3d9c-d121-469c-9a55-3e57392a1855" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607788 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeaa3d9c-d121-469c-9a55-3e57392a1855" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607804 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacf64e7-5435-4590-a8c1-7f67bd61d9cc" containerName="payload-pre-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607810 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacf64e7-5435-4590-a8c1-7f67bd61d9cc" containerName="payload-pre-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607818 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e57d02-10f0-43d9-ae3e-a3025ae00431" containerName="payload-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607823 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e57d02-10f0-43d9-ae3e-a3025ae00431" containerName="payload-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607832 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57af45fb-aa04-4521-8273-40271aa49356" containerName="manager" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607837 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="57af45fb-aa04-4521-8273-40271aa49356" containerName="manager" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607843 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1223d6-f0d2-4926-8ce8-d1bead24229d" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607848 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1223d6-f0d2-4926-8ce8-d1bead24229d" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607857 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb81ed94-3f6f-4698-bca6-757a4f286d50" containerName="payload-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607862 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb81ed94-3f6f-4698-bca6-757a4f286d50" containerName="payload-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607873 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e957f88c-b110-42d2-8425-64145a2bfd4c" containerName="payload-pre-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607878 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e957f88c-b110-42d2-8425-64145a2bfd4c" containerName="payload-pre-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607884 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d776d4c8-b1c2-4b8a-9188-eb63796cdce1" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607889 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d776d4c8-b1c2-4b8a-9188-eb63796cdce1" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607896 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe34356-d124-4917-b63a-87eb926021aa" containerName="payload-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607901 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe34356-d124-4917-b63a-87eb926021aa" containerName="payload-processing" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607952 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d776d4c8-b1c2-4b8a-9188-eb63796cdce1" containerName="maas-api" Sep 18 20:37:37.607954 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607961 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e57d02-10f0-43d9-ae3e-a3025ae00431" containerName="payload-processing" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607968 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="57af45fb-aa04-4521-8273-40271aa49356" containerName="manager" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607974 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb81ed94-3f6f-4698-bca6-757a4f286d50" containerName="payload-processing" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607979 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe34356-d124-4917-b63a-87eb926021aa" containerName="payload-processing" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607984 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeaa3d9c-d121-469c-9a55-3e57392a1855" containerName="maas-api" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607990 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacf64e7-5435-4590-a8c1-7f67bd61d9cc" containerName="payload-pre-processing" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.607995 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e957f88c-b110-42d2-8425-64145a2bfd4c" containerName="payload-pre-processing" Sep 18 20:37:37.608613 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.608001 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1223d6-f0d2-4926-8ce8-d1bead24229d" containerName="maas-api" Sep 18 20:37:37.612383 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.612366 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.616716 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.616692 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 20:37:37.616866 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.616773 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 20:37:37.616967 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.616693 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h6cq7\"" Sep 18 20:37:37.617203 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.617182 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 20:37:37.619730 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.619707 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-tdpgp"] Sep 18 20:37:37.757621 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.757577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-metrics-certs\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.757813 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.757629 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-webhook-cert\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.757813 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.757694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5np4\" (UniqueName: \"kubernetes.io/projected/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-kube-api-access-v5np4\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.858521 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.858423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-metrics-certs\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.858521 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.858475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-webhook-cert\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.858762 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.858532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5np4\" (UniqueName: \"kubernetes.io/projected/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-kube-api-access-v5np4\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.861081 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.861054 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-webhook-cert\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.861176 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.861099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-metrics-certs\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.866659 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.866640 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5np4\" (UniqueName: \"kubernetes.io/projected/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-kube-api-access-v5np4\") pod \"maas-controller-fb768dd77-tdpgp\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:37.923435 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:37.923401 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:38.194561 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:38.194531 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-tdpgp"] Sep 18 20:37:38.196190 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:37:38.196162 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7965c3_fddc_4595_9d94_c2cc02fc0e1a.slice/crio-16e6a12f65c20e081cac4779fb09dc964a8398ef34aa9a1628ea609b022fad28 WatchSource:0}: Error finding container 16e6a12f65c20e081cac4779fb09dc964a8398ef34aa9a1628ea609b022fad28: Status 404 returned error can't find the container with id 16e6a12f65c20e081cac4779fb09dc964a8398ef34aa9a1628ea609b022fad28 Sep 18 20:37:39.112578 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:39.112545 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-tdpgp" event={"ID":"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a","Type":"ContainerStarted","Data":"eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f"} Sep 18 20:37:39.113106 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:39.112603 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-tdpgp" event={"ID":"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a","Type":"ContainerStarted","Data":"16e6a12f65c20e081cac4779fb09dc964a8398ef34aa9a1628ea609b022fad28"} Sep 18 20:37:39.113106 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:39.112645 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:39.129658 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:39.129613 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fb768dd77-tdpgp" podStartSLOduration=1.839028229 podStartE2EDuration="2.129597353s" totalImagesPullingTime="290.569124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:37:39.128862782 +0000 UTC m=+2178.179223224" watchObservedRunningTime="2026-09-18 20:37:39.129597353 +0000 UTC m=+2178.179957801" Sep 18 20:37:50.120223 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:50.120193 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:59.468497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.468462 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fb768dd77-tdpgp"] Sep 18 20:37:59.468923 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.468666 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fb768dd77-tdpgp" podUID="fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" containerName="manager" containerID="cri-o://eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f" gracePeriod=30 Sep 18 20:37:59.700043 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.700020 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:37:59.840357 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.840268 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-metrics-certs\") pod \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " Sep 18 20:37:59.840357 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.840317 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-webhook-cert\") pod \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " Sep 18 20:37:59.840574 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.840359 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5np4\" (UniqueName: \"kubernetes.io/projected/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-kube-api-access-v5np4\") pod \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\" (UID: \"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a\") " Sep 18 20:37:59.842289 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.842240 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" (UID: "fc7965c3-fddc-4595-9d94-c2cc02fc0e1a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:37:59.842289 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.842250 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" (UID: "fc7965c3-fddc-4595-9d94-c2cc02fc0e1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:37:59.842443 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.842335 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-kube-api-access-v5np4" (OuterVolumeSpecName: "kube-api-access-v5np4") pod "fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" (UID: "fc7965c3-fddc-4595-9d94-c2cc02fc0e1a"). InnerVolumeSpecName "kube-api-access-v5np4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:37:59.941637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.941609 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:37:59.941637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.941631 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-webhook-cert\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:37:59.941637 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:37:59.941641 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5np4\" (UniqueName: \"kubernetes.io/projected/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a-kube-api-access-v5np4\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:38:00.213998 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.213963 2559 generic.go:356] "Generic (PLEG): container finished" podID="fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" containerID="eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f" exitCode=0 Sep 18 20:38:00.214189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.214005 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-tdpgp" Sep 18 20:38:00.214189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.214049 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-tdpgp" event={"ID":"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a","Type":"ContainerDied","Data":"eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f"} Sep 18 20:38:00.214189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.214095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-tdpgp" event={"ID":"fc7965c3-fddc-4595-9d94-c2cc02fc0e1a","Type":"ContainerDied","Data":"16e6a12f65c20e081cac4779fb09dc964a8398ef34aa9a1628ea609b022fad28"} Sep 18 20:38:00.214189 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.214119 2559 scope.go:117] "RemoveContainer" containerID="eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f" Sep 18 20:38:00.222987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.222968 2559 scope.go:117] "RemoveContainer" containerID="eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f" Sep 18 20:38:00.223246 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:38:00.223228 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f\": container with ID starting with eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f not found: ID does not exist" containerID="eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f" Sep 18 20:38:00.223318 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.223254 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f"} err="failed to get container status \"eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f\": rpc error: code = NotFound desc = could not find container \"eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f\": container with ID starting with eb0b6126993e704220484b5c092bd66fe45eb7e133e7226699b1d5e34eb60e6f not found: ID does not exist" Sep 18 20:38:00.235000 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.234954 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fb768dd77-tdpgp"] Sep 18 20:38:00.236524 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.236507 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fb768dd77-tdpgp"] Sep 18 20:38:00.992945 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.992915 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fb768dd77-c8v7m"] Sep 18 20:38:00.993483 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.993463 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" containerName="manager" Sep 18 20:38:00.993483 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.993485 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" containerName="manager" Sep 18 20:38:00.993683 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.993569 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" containerName="manager" Sep 18 20:38:00.997973 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:00.997951 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.001831 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.001805 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 20:38:01.001950 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.001931 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 20:38:01.002023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.001958 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 20:38:01.002197 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.002178 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h6cq7\"" Sep 18 20:38:01.004298 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.004249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-c8v7m"] Sep 18 20:38:01.150525 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.150494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-metrics-certs\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.150698 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.150544 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh88\" (UniqueName: \"kubernetes.io/projected/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-kube-api-access-hgh88\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.150698 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.150565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-webhook-cert\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.183554 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.183520 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7965c3-fddc-4595-9d94-c2cc02fc0e1a" path="/var/lib/kubelet/pods/fc7965c3-fddc-4595-9d94-c2cc02fc0e1a/volumes" Sep 18 20:38:01.251166 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.251142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh88\" (UniqueName: \"kubernetes.io/projected/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-kube-api-access-hgh88\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.251166 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.251167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-webhook-cert\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.251359 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.251218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-metrics-certs\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.253621 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.253602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-webhook-cert\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.253735 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.253637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-metrics-certs\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.259203 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.259185 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh88\" (UniqueName: \"kubernetes.io/projected/b0b66956-0c2d-4bf0-8181-ecb7d63dedce-kube-api-access-hgh88\") pod \"maas-controller-fb768dd77-c8v7m\" (UID: \"b0b66956-0c2d-4bf0-8181-ecb7d63dedce\") " pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.309166 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.309133 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:01.588344 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:01.582913 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fb768dd77-c8v7m"] Sep 18 20:38:01.589918 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:38:01.589892 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b66956_0c2d_4bf0_8181_ecb7d63dedce.slice/crio-3ef1903883c16c9f1362c3a818f349416306937025487232943fdbd886785591 WatchSource:0}: Error finding container 3ef1903883c16c9f1362c3a818f349416306937025487232943fdbd886785591: Status 404 returned error can't find the container with id 3ef1903883c16c9f1362c3a818f349416306937025487232943fdbd886785591 Sep 18 20:38:02.227293 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:02.227240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-c8v7m" event={"ID":"b0b66956-0c2d-4bf0-8181-ecb7d63dedce","Type":"ContainerStarted","Data":"61bbb5e902fd754320deeb34410fe4c98a524e5380e38c48d20b3753d1ecea2a"} Sep 18 20:38:02.227692 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:02.227304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fb768dd77-c8v7m" event={"ID":"b0b66956-0c2d-4bf0-8181-ecb7d63dedce","Type":"ContainerStarted","Data":"3ef1903883c16c9f1362c3a818f349416306937025487232943fdbd886785591"} Sep 18 20:38:02.227692 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:02.227351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:38:02.245399 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:02.245342 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fb768dd77-c8v7m" podStartSLOduration=1.745141123 podStartE2EDuration="2.245323259s" totalImagesPullingTime="500.182136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:38:02.243682867 +0000 UTC m=+2201.294043316" watchObservedRunningTime="2026-09-18 20:38:02.245323259 +0000 UTC m=+2201.295683706" Sep 18 20:38:13.234102 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:38:13.234070 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fb768dd77-c8v7m" Sep 18 20:41:23.335082 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:41:23.335053 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:41:23.339836 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:41:23.339813 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:42:32.520625 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.520592 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g"] Sep 18 20:42:32.524009 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.523979 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.536796 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.536763 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g"] Sep 18 20:42:32.607487 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.607447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-metrics-certs\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.607688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.607505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-openshift-service-ca\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.607688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.607545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-maas-api-tls\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.607688 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.607571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgll\" (UniqueName: \"kubernetes.io/projected/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-kube-api-access-scgll\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.708061 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.708019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgll\" (UniqueName: \"kubernetes.io/projected/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-kube-api-access-scgll\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.708260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.708182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-metrics-certs\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.708260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.708230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-openshift-service-ca\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.708260 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.708252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-maas-api-tls\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.709023 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.708999 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-openshift-service-ca\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.710830 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.710806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-metrics-certs\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.711215 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.711190 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-maas-api-tls\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.716972 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.716945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgll\" (UniqueName: \"kubernetes.io/projected/c2e4d737-7eeb-4551-a728-f2ae6e72d26d-kube-api-access-scgll\") pod \"maas-api-86d9ddc875-8cx4g\" (UID: \"c2e4d737-7eeb-4551-a728-f2ae6e72d26d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:32.834942 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:32.834858 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:33.110791 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:33.110730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g"] Sep 18 20:42:33.113103 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:42:33.113074 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e4d737_7eeb_4551_a728_f2ae6e72d26d.slice/crio-400fb8d2026cbc22eb06bccb97b186c930e0cb1203fc5fb1ad13674deb5e1ff2 WatchSource:0}: Error finding container 400fb8d2026cbc22eb06bccb97b186c930e0cb1203fc5fb1ad13674deb5e1ff2: Status 404 returned error can't find the container with id 400fb8d2026cbc22eb06bccb97b186c930e0cb1203fc5fb1ad13674deb5e1ff2 Sep 18 20:42:33.114419 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:33.114404 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 20:42:33.508712 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:33.508675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" event={"ID":"c2e4d737-7eeb-4551-a728-f2ae6e72d26d","Type":"ContainerStarted","Data":"81e13613abfb2f8baaceaaebcf90c14c8cc6013fbd26cd9fc9e51977ecabdd34"} Sep 18 20:42:33.508909 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:33.508736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" event={"ID":"c2e4d737-7eeb-4551-a728-f2ae6e72d26d","Type":"ContainerStarted","Data":"400fb8d2026cbc22eb06bccb97b186c930e0cb1203fc5fb1ad13674deb5e1ff2"} Sep 18 20:42:33.508909 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:33.508792 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:33.529233 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:33.529175 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" podStartSLOduration=1.2223714079999999 podStartE2EDuration="1.529157877s" totalImagesPullingTime="306.786469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:42:33.52782495 +0000 UTC m=+2472.578185399" watchObservedRunningTime="2026-09-18 20:42:33.529157877 +0000 UTC m=+2472.579518360" Sep 18 20:42:39.519239 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:39.519212 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-8cx4g" Sep 18 20:42:47.253750 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:47.253714 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52"] Sep 18 20:42:47.254158 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:47.253914 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" podUID="ab6af800-ebb0-4a94-8174-7b2f83ea7b15" containerName="maas-api" containerID="cri-o://9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c" gracePeriod=30 Sep 18 20:42:48.497445 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.497420 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:42:48.541548 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.541523 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs\") pod \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " Sep 18 20:42:48.541709 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.541590 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52gnd\" (UniqueName: \"kubernetes.io/projected/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-kube-api-access-52gnd\") pod \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " Sep 18 20:42:48.541709 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.541614 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-maas-api-tls\") pod \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " Sep 18 20:42:48.541709 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.541633 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-openshift-service-ca\") pod \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\" (UID: \"ab6af800-ebb0-4a94-8174-7b2f83ea7b15\") " Sep 18 20:42:48.542064 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.542036 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab6af800-ebb0-4a94-8174-7b2f83ea7b15" (UID: "ab6af800-ebb0-4a94-8174-7b2f83ea7b15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 20:42:48.543606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.543577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab6af800-ebb0-4a94-8174-7b2f83ea7b15" (UID: "ab6af800-ebb0-4a94-8174-7b2f83ea7b15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:42:48.543700 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.543664 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab6af800-ebb0-4a94-8174-7b2f83ea7b15" (UID: "ab6af800-ebb0-4a94-8174-7b2f83ea7b15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 20:42:48.543700 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.543685 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-kube-api-access-52gnd" (OuterVolumeSpecName: "kube-api-access-52gnd") pod "ab6af800-ebb0-4a94-8174-7b2f83ea7b15" (UID: "ab6af800-ebb0-4a94-8174-7b2f83ea7b15"). InnerVolumeSpecName "kube-api-access-52gnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 20:42:48.580584 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.580555 2559 generic.go:356] "Generic (PLEG): container finished" podID="ab6af800-ebb0-4a94-8174-7b2f83ea7b15" containerID="9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c" exitCode=0 Sep 18 20:42:48.580717 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.580600 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" Sep 18 20:42:48.580717 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.580644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" event={"ID":"ab6af800-ebb0-4a94-8174-7b2f83ea7b15","Type":"ContainerDied","Data":"9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c"} Sep 18 20:42:48.580798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.580735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52" event={"ID":"ab6af800-ebb0-4a94-8174-7b2f83ea7b15","Type":"ContainerDied","Data":"d60bc2ec41afea99b0f90c7d528e91b344384e90bbe5620af22069d469e5f016"} Sep 18 20:42:48.580798 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.580759 2559 scope.go:117] "RemoveContainer" containerID="9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c" Sep 18 20:42:48.590112 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.590097 2559 scope.go:117] "RemoveContainer" containerID="9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c" Sep 18 20:42:48.590408 ip-10-0-135-232 kubenswrapper[2559]: E0918 20:42:48.590388 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c\": container with ID starting with 9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c not found: ID does not exist" containerID="9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c" Sep 18 20:42:48.590549 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.590415 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c"} err="failed to get container status \"9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c\": rpc error: code = NotFound desc = could not find container \"9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c\": container with ID starting with 9ef6fcc706ce0308235a097a9d20f04b9ad443e0ae7979bf5fe76107cd205d5c not found: ID does not exist" Sep 18 20:42:48.602770 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.602705 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52"] Sep 18 20:42:48.604542 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.604524 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-x7m52"] Sep 18 20:42:48.642679 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.642647 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-metrics-certs\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:42:48.642679 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.642675 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52gnd\" (UniqueName: \"kubernetes.io/projected/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-kube-api-access-52gnd\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:42:48.642679 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.642686 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-maas-api-tls\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:42:48.642909 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:48.642697 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab6af800-ebb0-4a94-8174-7b2f83ea7b15-openshift-service-ca\") on node \"ip-10-0-135-232.ec2.internal\" DevicePath \"\"" Sep 18 20:42:49.183873 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:42:49.183839 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6af800-ebb0-4a94-8174-7b2f83ea7b15" path="/var/lib/kubelet/pods/ab6af800-ebb0-4a94-8174-7b2f83ea7b15/volumes" Sep 18 20:43:02.856921 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:02.856891 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8fq8w_c8bad512-0d51-4c8d-a288-bcc8c17049e8/global-pull-secret-syncer/0.log" Sep 18 20:43:03.087777 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:03.087746 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xlkx8_2e878eff-e7a1-43a2-bf2c-660de829e65f/konnectivity-agent/0.log" Sep 18 20:43:03.221606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:03.221527 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-232.ec2.internal_5ffd3a0f2bc2beb85186ad42f65a5756/haproxy/0.log" Sep 18 20:43:07.546705 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:07.546617 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-qwthl_b1b549ca-85b2-4e54-8c2b-6a7ac4052f02/manager/0.log" Sep 18 20:43:07.721732 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:07.721702 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-wbbj2_7452724f-7913-4a39-aa7b-d20b755b8c1d/manager/0.log" Sep 18 20:43:09.238333 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.238304 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/alertmanager/0.log" Sep 18 20:43:09.268442 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.268420 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/config-reloader/0.log" Sep 18 20:43:09.289497 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.289477 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/kube-rbac-proxy-web/0.log" Sep 18 20:43:09.324415 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.324393 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/kube-rbac-proxy/0.log" Sep 18 20:43:09.347817 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.347797 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/kube-rbac-proxy-metric/0.log" Sep 18 20:43:09.366718 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.366701 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/prom-label-proxy/0.log" Sep 18 20:43:09.390020 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.390001 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c7194915-7361-4d15-8e63-4b9243aaa185/init-config-reloader/0.log" Sep 18 20:43:09.548722 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.548647 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-8n7ns_1b422ad9-fc08-447e-b054-fb9513eaa935/monitoring-plugin/0.log" Sep 18 20:43:09.576050 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.576025 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wgnp_f3debf0f-62b6-44b4-824b-506fab50d6a4/node-exporter/0.log" Sep 18 20:43:09.597155 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.597134 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wgnp_f3debf0f-62b6-44b4-824b-506fab50d6a4/kube-rbac-proxy/0.log" Sep 18 20:43:09.616338 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:09.616315 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wgnp_f3debf0f-62b6-44b4-824b-506fab50d6a4/init-textfile/0.log" Sep 18 20:43:10.239603 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.239528 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-w6gr7_2e7deb29-dc62-4291-b7e7-1c92b0a3e48a/prometheus-operator-admission-webhook/0.log" Sep 18 20:43:10.363987 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.363958 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d88f99d6d-zk9lv_ddbf0fba-4548-49d5-b602-1522ac7872fb/thanos-query/0.log" Sep 18 20:43:10.384141 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.384114 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d88f99d6d-zk9lv_ddbf0fba-4548-49d5-b602-1522ac7872fb/kube-rbac-proxy-web/0.log" Sep 18 20:43:10.406808 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.406788 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d88f99d6d-zk9lv_ddbf0fba-4548-49d5-b602-1522ac7872fb/kube-rbac-proxy/0.log" Sep 18 20:43:10.428693 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.428673 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d88f99d6d-zk9lv_ddbf0fba-4548-49d5-b602-1522ac7872fb/prom-label-proxy/0.log" Sep 18 20:43:10.447753 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.447732 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d88f99d6d-zk9lv_ddbf0fba-4548-49d5-b602-1522ac7872fb/kube-rbac-proxy-rules/0.log" Sep 18 20:43:10.468870 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:10.468850 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d88f99d6d-zk9lv_ddbf0fba-4548-49d5-b602-1522ac7872fb/kube-rbac-proxy-metrics/0.log" Sep 18 20:43:12.660702 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.660674 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf"] Sep 18 20:43:12.661074 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.661046 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6af800-ebb0-4a94-8174-7b2f83ea7b15" containerName="maas-api" Sep 18 20:43:12.661074 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.661057 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6af800-ebb0-4a94-8174-7b2f83ea7b15" containerName="maas-api" Sep 18 20:43:12.661150 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.661115 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab6af800-ebb0-4a94-8174-7b2f83ea7b15" containerName="maas-api" Sep 18 20:43:12.664493 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.664478 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.666986 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.666966 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jwttx\"/\"default-dockercfg-hp79t\"" Sep 18 20:43:12.667116 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.667030 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jwttx\"/\"openshift-service-ca.crt\"" Sep 18 20:43:12.668142 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.668125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jwttx\"/\"kube-root-ca.crt\"" Sep 18 20:43:12.673423 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.673402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf"] Sep 18 20:43:12.755822 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.755793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-proc\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.755960 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.755829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-sys\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.755960 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.755861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7xq\" (UniqueName: \"kubernetes.io/projected/d4eb2c43-c941-47d9-9036-6b48199f1761-kube-api-access-gb7xq\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.756046 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.755958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-podres\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.756046 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.755998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-lib-modules\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856446 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856411 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-podres\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-lib-modules\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-proc\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-sys\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7xq\" (UniqueName: \"kubernetes.io/projected/d4eb2c43-c941-47d9-9036-6b48199f1761-kube-api-access-gb7xq\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856606 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-podres\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856797 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-proc\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856797 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-lib-modules\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.856797 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.856627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4eb2c43-c941-47d9-9036-6b48199f1761-sys\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.864168 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.864145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7xq\" (UniqueName: \"kubernetes.io/projected/d4eb2c43-c941-47d9-9036-6b48199f1761-kube-api-access-gb7xq\") pod \"perf-node-gather-daemonset-tlwlf\" (UID: \"d4eb2c43-c941-47d9-9036-6b48199f1761\") " pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:12.975087 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:12.975018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:13.247363 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:13.247336 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf"] Sep 18 20:43:13.248927 ip-10-0-135-232 kubenswrapper[2559]: W0918 20:43:13.248896 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4eb2c43_c941_47d9_9036_6b48199f1761.slice/crio-bc7fd6893f14afa01f4d3f735dc5a84e32cc5bec6b127b6cf15724e76af6819d WatchSource:0}: Error finding container bc7fd6893f14afa01f4d3f735dc5a84e32cc5bec6b127b6cf15724e76af6819d: Status 404 returned error can't find the container with id bc7fd6893f14afa01f4d3f735dc5a84e32cc5bec6b127b6cf15724e76af6819d Sep 18 20:43:13.705431 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:13.705400 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" event={"ID":"d4eb2c43-c941-47d9-9036-6b48199f1761","Type":"ContainerStarted","Data":"294b1412cce29946470d1345598275f8f728ef5138c6304c277e52288cf0ff2f"} Sep 18 20:43:13.705823 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:13.705453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:13.705823 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:13.705471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" event={"ID":"d4eb2c43-c941-47d9-9036-6b48199f1761","Type":"ContainerStarted","Data":"bc7fd6893f14afa01f4d3f735dc5a84e32cc5bec6b127b6cf15724e76af6819d"} Sep 18 20:43:13.721146 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:13.721094 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" podStartSLOduration=1.721080407 podStartE2EDuration="1.721080407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:43:13.719580904 +0000 UTC m=+2512.769941353" watchObservedRunningTime="2026-09-18 20:43:13.721080407 +0000 UTC m=+2512.771440854" Sep 18 20:43:14.764975 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:14.764945 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-krmvc_bb2e0673-cd29-40b6-b85c-15851a3174e6/dns/0.log" Sep 18 20:43:14.785571 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:14.785548 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-krmvc_bb2e0673-cd29-40b6-b85c-15851a3174e6/kube-rbac-proxy/0.log" Sep 18 20:43:14.949059 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:14.949030 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgcsb_2568c6a7-7806-4cdf-86cb-989ac8ea8e1b/dns-node-resolver/0.log" Sep 18 20:43:15.528462 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:15.528439 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4t6zl_030ae6db-d323-4dcc-8242-6750bb91b611/node-ca/0.log" Sep 18 20:43:16.844034 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:16.844009 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bw7lg_3eea43bd-e794-42d0-a8b9-aaddb194dcf8/discovery/0.log" Sep 18 20:43:16.863157 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:16.863136 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b8956864-gk67t_80355379-fa48-4644-aea3-daef56ee1aa8/kube-auth-proxy/0.log" Sep 18 20:43:16.959515 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:16.959499 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:43:17.845270 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:17.845241 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdqjn_574561ec-4096-443d-8095-3f7d94cbd239/serve-healthcheck-canary/0.log" Sep 18 20:43:18.364289 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:18.364253 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7plkl_a07ae61c-d277-4b52-8026-9f49b7fb2deb/kube-rbac-proxy/0.log" Sep 18 20:43:18.385108 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:18.385084 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7plkl_a07ae61c-d277-4b52-8026-9f49b7fb2deb/exporter/0.log" Sep 18 20:43:18.405192 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:18.405171 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7plkl_a07ae61c-d277-4b52-8026-9f49b7fb2deb/extractor/0.log" Sep 18 20:43:19.720607 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:19.720580 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jwttx/perf-node-gather-daemonset-tlwlf" Sep 18 20:43:20.512848 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:20.512821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-587bdb7f54-cgd9v_3cbd8d09-10b4-4080-9a86-897781329049/manager/0.log" Sep 18 20:43:21.001749 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:21.001720 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-fb768dd77-c8v7m_b0b66956-0c2d-4bf0-8181-ecb7d63dedce/manager/0.log" Sep 18 20:43:21.019474 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:21.019451 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wwh6j_60986192-d301-4e5a-ad99-3c63d1667c85/server/0.log" Sep 18 20:43:21.094581 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:21.094558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h8d5d_23612048-c42d-4dc2-96c2-5c10e94b7305/manager/0.log" Sep 18 20:43:21.253156 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:21.253087 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-668c7cd96d-swxwf_f584d8bd-54fa-4dde-8ae4-2633a9f36ee8/manager/0.log" Sep 18 20:43:21.267820 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:21.267802 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-668c7cd96d-swxwf_f584d8bd-54fa-4dde-8ae4-2633a9f36ee8/copy-manifests/0.log" Sep 18 20:43:22.561143 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:22.561110 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-s2v24_44549c80-4b4f-41e0-a1aa-53f1a130aec6/openshift-lws-operator/0.log" Sep 18 20:43:22.907854 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:22.907827 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bw7lg_3eea43bd-e794-42d0-a8b9-aaddb194dcf8/discovery/0.log" Sep 18 20:43:22.920433 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:22.920410 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b8956864-gk67t_80355379-fa48-4644-aea3-daef56ee1aa8/kube-auth-proxy/0.log" Sep 18 20:43:22.993256 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:22.993232 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:43:24.887385 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:24.887350 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bw7lg_3eea43bd-e794-42d0-a8b9-aaddb194dcf8/discovery/0.log" Sep 18 20:43:24.906387 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:24.906359 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b8956864-gk67t_80355379-fa48-4644-aea3-daef56ee1aa8/kube-auth-proxy/0.log" Sep 18 20:43:24.974713 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:24.974692 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-7wlzn_903c000c-3e8a-4eea-8ef7-5d3d52496350/payload-processing/0.log" Sep 18 20:43:28.847238 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:28.847201 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-65zh9_be325712-4ebe-4aba-ba1a-62ffa8b9d0d7/kube-multus/0.log" Sep 18 20:43:29.089207 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.089180 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/kube-multus-additional-cni-plugins/0.log" Sep 18 20:43:29.108455 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.108434 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/egress-router-binary-copy/0.log" Sep 18 20:43:29.127589 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.127568 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/cni-plugins/0.log" Sep 18 20:43:29.146658 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.146638 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/bond-cni-plugin/0.log" Sep 18 20:43:29.169975 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.169958 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/routeoverride-cni/0.log" Sep 18 20:43:29.193334 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.193317 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/whereabouts-cni-bincopy/0.log" Sep 18 20:43:29.214003 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.213982 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgqn7_d414546a-d81e-4267-9179-a3f81929420b/whereabouts-cni/0.log" Sep 18 20:43:29.480881 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.480812 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7m46g_6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472/network-metrics-daemon/0.log" Sep 18 20:43:29.497022 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:29.496995 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7m46g_6a32d2ac-9ee8-4f19-8fc3-3b18d4a29472/kube-rbac-proxy/0.log" Sep 18 20:43:30.563431 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.563402 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-controller/0.log" Sep 18 20:43:30.578936 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.578915 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/0.log" Sep 18 20:43:30.579074 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.579062 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovn-acl-logging/1.log" Sep 18 20:43:30.597354 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.597332 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/kube-rbac-proxy-node/0.log" Sep 18 20:43:30.618153 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.618131 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/kube-rbac-proxy-ovn-metrics/0.log" Sep 18 20:43:30.636842 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.636822 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/northd/0.log" Sep 18 20:43:30.658701 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.658683 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/nbdb/0.log" Sep 18 20:43:30.677801 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.677782 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/sbdb/0.log" Sep 18 20:43:30.733514 ip-10-0-135-232 kubenswrapper[2559]: I0918 20:43:30.733498 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qztvj_2c16f313-cb28-4202-a0a9-3eef01008a52/ovnkube-controller/0.log"