Sep 06 13:57:30.608256 ip-10-0-141-9 systemd[1]: Starting Kubernetes Kubelet... Sep 06 13:57:31.058926 ip-10-0-141-9 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 13:57:31.058926 ip-10-0-141-9 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 06 13:57:31.058926 ip-10-0-141-9 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 13:57:31.058926 ip-10-0-141-9 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 06 13:57:31.058926 ip-10-0-141-9 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 13:57:31.059940 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.059825 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 06 13:57:31.064088 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064064 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:31.064088 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064088 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064093 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064099 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064103 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064108 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064119 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064124 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064128 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064132 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064137 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064141 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064145 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064149 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064153 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064156 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064163 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064167 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064171 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:31.064167 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064175 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064179 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064183 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064187 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064191 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064194 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064198 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064202 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064205 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064209 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064212 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064216 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064219 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064223 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064227 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064230 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064234 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064237 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064241 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064244 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:31.064594 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064251 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064256 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064261 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064265 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064269 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064274 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064277 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064281 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064285 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064289 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064293 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064297 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064301 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064305 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064309 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064313 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064317 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064321 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064325 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:31.065130 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064329 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064332 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064336 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064339 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064343 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064347 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064351 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064354 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064358 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064363 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064368 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064373 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064376 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064381 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064385 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064388 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064392 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064396 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064400 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064405 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:31.065593 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064409 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064413 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064417 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064421 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064426 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064431 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064435 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064439 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064442 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064446 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064450 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064453 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064457 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064461 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064464 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064468 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064472 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064475 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064479 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064482 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:31.066083 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.064486 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065480 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065490 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065495 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065499 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065503 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065508 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065512 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065516 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065521 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065524 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065528 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065533 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065537 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065542 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065546 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065549 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065553 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065557 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:31.066568 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065561 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065566 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065571 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065574 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065578 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065581 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065585 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065589 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065593 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065597 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065601 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065604 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065608 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065612 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065616 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065620 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065624 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065627 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065631 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:31.067040 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065638 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065644 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065648 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065653 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065658 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065663 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065669 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065674 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065678 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065682 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065686 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065690 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065693 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065697 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065700 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065704 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065707 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065711 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065715 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:31.067511 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065718 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065722 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065725 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065729 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065732 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065736 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065740 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065743 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065747 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065750 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065754 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065758 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065762 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065767 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065771 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065774 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065778 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065783 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065787 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065791 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:31.067992 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065796 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065801 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065805 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065809 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065813 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065817 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065820 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065824 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065828 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065832 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065836 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065841 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065844 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065848 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065852 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065856 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065860 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065864 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065867 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:31.068481 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065871 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065874 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065878 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.065882 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066539 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066547 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066553 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066559 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066563 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066568 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066572 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066577 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066582 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066586 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066591 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066595 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066598 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066602 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066605 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:31.068940 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066610 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066613 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066617 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066620 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066655 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066661 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066666 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066669 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066673 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066677 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066682 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066686 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066690 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066693 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066697 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066700 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066704 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066708 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066712 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:31.069416 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066716 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066722 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066728 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066732 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066736 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066740 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066744 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066747 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066751 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066756 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066760 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066763 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066766 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066770 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066773 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066777 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066781 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066785 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066789 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:31.069877 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066792 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066796 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066799 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066803 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066807 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066810 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066814 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066817 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066821 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066824 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066828 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066832 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066836 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066840 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066843 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066847 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066854 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066859 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066864 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:31.070381 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066868 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066871 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066876 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066880 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066884 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066901 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066906 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066910 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066914 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066918 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066922 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066926 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066929 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066932 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066936 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066940 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066944 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066947 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066951 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066955 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:31.070834 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066958 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066962 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066966 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066970 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066973 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066976 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.066980 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067074 2550 flags.go:64] FLAG: --address="0.0.0.0" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067122 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067132 2550 flags.go:64] FLAG: --anonymous-auth="true" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067139 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067145 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067150 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067156 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067163 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067167 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067171 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067177 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067182 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067187 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067191 2550 flags.go:64] FLAG: --cgroup-root="" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067196 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067200 2550 flags.go:64] FLAG: --client-ca-file="" Sep 06 13:57:31.071323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067204 2550 flags.go:64] FLAG: --cloud-provider="external" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067208 2550 flags.go:64] FLAG: --cluster-dns="[]" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067214 2550 flags.go:64] FLAG: --cluster-domain="" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067218 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067223 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067228 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067235 2550 flags.go:64] FLAG: --container-log-max-files="5" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067241 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067244 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067249 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067254 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067258 2550 flags.go:64] FLAG: --contention-profiling="false" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067261 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067266 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067271 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067276 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067283 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067287 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067292 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067298 2550 flags.go:64] FLAG: --enable-load-reader="false" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067303 2550 flags.go:64] FLAG: --enable-server="true" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067308 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067315 2550 flags.go:64] FLAG: --event-burst="100" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067319 2550 flags.go:64] FLAG: --event-qps="50" Sep 06 13:57:31.071932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067324 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067328 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067333 2550 flags.go:64] FLAG: --eviction-hard="" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067339 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067344 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067348 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067353 2550 flags.go:64] FLAG: --eviction-soft="" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067357 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067362 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067366 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067370 2550 flags.go:64] FLAG: --experimental-mounter-path="" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067374 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067379 2550 flags.go:64] FLAG: --fail-swap-on="true" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067383 2550 flags.go:64] FLAG: --feature-gates="" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067390 2550 flags.go:64] FLAG: --file-check-frequency="20s" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067394 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067398 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067403 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067408 2550 flags.go:64] FLAG: --healthz-port="10248" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067412 2550 flags.go:64] FLAG: --help="false" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067416 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067421 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067425 2550 flags.go:64] FLAG: --http-check-frequency="20s" Sep 06 13:57:31.072515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067430 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067435 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067439 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067443 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067449 2550 flags.go:64] FLAG: --image-service-endpoint="" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067454 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067458 2550 flags.go:64] FLAG: --kube-api-burst="100" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067463 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067467 2550 flags.go:64] FLAG: --kube-api-qps="50" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067472 2550 flags.go:64] FLAG: --kube-reserved="" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067476 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067481 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067485 2550 flags.go:64] FLAG: --kubelet-cgroups="" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067490 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067494 2550 flags.go:64] FLAG: --lock-file="" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067499 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067504 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067508 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067515 2550 flags.go:64] FLAG: --log-json-split-stream="false" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067520 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067525 2550 flags.go:64] FLAG: --log-text-split-stream="false" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067529 2550 flags.go:64] FLAG: --logging-format="text" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067534 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067540 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 06 13:57:31.073136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067545 2550 flags.go:64] FLAG: --manifest-url="" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067549 2550 flags.go:64] FLAG: --manifest-url-header="" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067555 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067559 2550 flags.go:64] FLAG: --max-open-files="1000000" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067565 2550 flags.go:64] FLAG: --max-pods="110" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067570 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067574 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067579 2550 flags.go:64] FLAG: --memory-manager-policy="None" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067583 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067587 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067591 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067596 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067608 2550 flags.go:64] FLAG: --node-status-max-images="50" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067612 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067617 2550 flags.go:64] FLAG: --oom-score-adj="-999" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067622 2550 flags.go:64] FLAG: --pod-cidr="" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067626 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067634 2550 flags.go:64] FLAG: --pod-manifest-path="" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067638 2550 flags.go:64] FLAG: --pod-max-pids="-1" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067643 2550 flags.go:64] FLAG: --pods-per-core="0" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067648 2550 flags.go:64] FLAG: --port="10250" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067652 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067658 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e7a79a89eacda8f9" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067663 2550 flags.go:64] FLAG: --qos-reserved="" Sep 06 13:57:31.073794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067667 2550 flags.go:64] FLAG: --read-only-port="10255" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067672 2550 flags.go:64] FLAG: --register-node="true" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067676 2550 flags.go:64] FLAG: --register-with-taints="" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067681 2550 flags.go:64] FLAG: --registry-burst="10" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067685 2550 flags.go:64] FLAG: --registry-qps="5" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067689 2550 flags.go:64] FLAG: --reserved-cpus="" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067693 2550 flags.go:64] FLAG: --reserved-memory="" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067699 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067704 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067709 2550 flags.go:64] FLAG: --rotate-certificates="false" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067713 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067718 2550 flags.go:64] FLAG: --runonce="false" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067722 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067727 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067731 2550 flags.go:64] FLAG: --seccomp-default="false" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067735 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067740 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067744 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067748 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067753 2550 flags.go:64] FLAG: --storage-driver-password="root" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067759 2550 flags.go:64] FLAG: --storage-driver-secure="false" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067763 2550 flags.go:64] FLAG: --storage-driver-table="stats" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067768 2550 flags.go:64] FLAG: --storage-driver-user="root" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067772 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067777 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 06 13:57:31.074394 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067781 2550 flags.go:64] FLAG: --system-cgroups="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067785 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067793 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067798 2550 flags.go:64] FLAG: --tls-cert-file="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067802 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067808 2550 flags.go:64] FLAG: --tls-min-version="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067813 2550 flags.go:64] FLAG: --tls-private-key-file="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067817 2550 flags.go:64] FLAG: --topology-manager-policy="none" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067821 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067826 2550 flags.go:64] FLAG: --topology-manager-scope="container" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067830 2550 flags.go:64] FLAG: --v="2" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067836 2550 flags.go:64] FLAG: --version="false" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067842 2550 flags.go:64] FLAG: --vmodule="" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067847 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.067852 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068039 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068045 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068049 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068053 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068057 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068061 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068064 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068068 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:31.075018 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068072 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068076 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068080 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068084 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068089 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068093 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068097 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068101 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068105 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068116 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068120 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068124 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068129 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068133 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068137 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068141 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068146 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068150 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068154 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068157 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:31.075609 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068161 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068165 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068168 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068172 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068177 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068180 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068184 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068188 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068191 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068195 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068199 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068203 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068206 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068210 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068213 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068217 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068222 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068226 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068229 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068233 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:31.076172 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068236 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068240 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068244 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068248 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068251 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068255 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068258 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068262 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068266 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068270 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068275 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068280 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068284 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068288 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068292 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068295 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068299 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068303 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068307 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:31.076670 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068310 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068314 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068318 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068321 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068324 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068328 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068331 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068335 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068339 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068344 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068347 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068350 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068354 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068360 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068366 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068369 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068373 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068376 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068380 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:31.077148 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068384 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068387 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068391 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068394 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068398 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068402 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068406 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068410 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068413 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068417 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068422 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068427 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.068431 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.069346 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.077473 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 06 13:57:31.077618 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.077492 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.077514 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.077524 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077583 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077588 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077591 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077594 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077598 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077600 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077603 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077607 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077610 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077613 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077616 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077619 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077622 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077625 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077628 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077631 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:31.078005 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077634 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077637 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077640 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077642 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077645 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077648 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077650 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077653 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077656 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077659 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077661 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077664 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077666 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077670 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077673 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077676 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077679 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077682 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077685 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077687 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:31.078465 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077690 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077693 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077696 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077698 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077701 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077703 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077706 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077708 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077711 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077714 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077716 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077719 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077722 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077725 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077728 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077730 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077734 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077738 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077742 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:31.079031 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077744 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077748 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077750 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077753 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077755 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077758 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077761 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077765 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077768 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077771 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077774 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077776 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077779 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077781 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077784 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077786 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077789 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077792 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077794 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077797 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:31.079486 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077799 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077802 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077805 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077807 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077810 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077813 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077815 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077818 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077820 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077822 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077825 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077828 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077830 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077834 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077838 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077841 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077844 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077847 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077850 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077853 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:31.079980 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077857 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077860 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077863 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.077867 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.077872 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078014 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078020 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078023 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078026 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078029 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078033 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078038 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078041 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078044 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 13:57:31.080475 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078047 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078049 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078052 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078055 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078057 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078060 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078062 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078065 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078068 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078071 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078073 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078076 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078078 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078081 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078083 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078087 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078090 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078094 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078097 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078100 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 13:57:31.080823 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078103 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078106 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078109 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078111 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078114 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078117 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078119 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078123 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078127 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078130 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078133 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078136 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078139 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078141 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078145 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078148 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078150 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078153 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078156 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 13:57:31.081350 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078159 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078162 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078165 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078168 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078171 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078173 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078176 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078179 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078182 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078184 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078187 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078190 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078193 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078197 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078199 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078202 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078204 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078208 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078210 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078213 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 13:57:31.081804 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078216 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078218 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078221 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078224 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078226 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078229 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078231 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078234 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078236 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078239 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078242 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078244 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078247 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078249 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078252 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078255 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078258 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078261 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078263 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078265 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 13:57:31.082318 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078268 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078271 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078273 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078276 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078279 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078282 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078285 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078288 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078290 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078293 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:31.078295 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.078301 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.078429 2550 server.go:962] "Client rotation is on, will bootstrap in background" Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.080963 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 06 13:57:31.082817 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.082089 2550 server.go:1019] "Starting client certificate rotation" Sep 06 13:57:31.083169 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.082188 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 13:57:31.083169 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.082232 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 13:57:31.085010 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.084992 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 13:57:31.086281 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.086217 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 13:57:31.098346 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.098317 2550 log.go:25] "Validated CRI v1 runtime API" Sep 06 13:57:31.104118 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.104086 2550 log.go:25] "Validated CRI v1 image API" Sep 06 13:57:31.105140 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.105114 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 06 13:57:31.108689 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.108660 2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81582360-3797-4cc1-86e6-67da2d74c60b:/dev/nvme0n1p3 8aabbd2a-541e-4173-a3fc-49c9800b85eb:/dev/nvme0n1p4] Sep 06 13:57:31.108762 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.108687 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 06 13:57:31.114482 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.114461 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 13:57:31.115027 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.114910 2550 manager.go:216] Machine: {Timestamp:2026-09-06 13:57:31.113461195 +0000 UTC m=+0.376905910 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101940 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21fbc7fd4b92f4149a2b647776c052 SystemUUID:ec21fbc7-fd4b-92f4-149a-2b647776c052 BootID:6c0a52bb-7d70-44c3-a3b6-c4889734be56 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:63:10:3a:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:63:10:3a:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:58:ed:f5:e1:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 06 13:57:31.115027 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.115023 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 06 13:57:31.115135 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.115125 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 06 13:57:31.118008 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.117975 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 06 13:57:31.118181 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.118011 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 06 13:57:31.118932 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.118920 2550 topology_manager.go:138] "Creating topology manager with none policy" Sep 06 13:57:31.118975 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.118934 2550 container_manager_linux.go:309] "Creating device plugin manager" Sep 06 13:57:31.118975 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.118943 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 13:57:31.120619 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.120603 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 13:57:31.122624 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.122607 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 06 13:57:31.127390 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127373 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 06 13:57:31.127526 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127515 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 06 13:57:31.127587 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127580 2550 kubelet.go:486] "Attempting to sync node with API server" Sep 06 13:57:31.127627 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127593 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 06 13:57:31.127627 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127612 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 06 13:57:31.127627 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127622 2550 kubelet.go:398] "Adding apiserver pod source" Sep 06 13:57:31.127727 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.127642 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 06 13:57:31.128090 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.128077 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 13:57:31.128151 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.128103 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 13:57:31.130580 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.130559 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 06 13:57:31.133575 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133556 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 06 13:57:31.133627 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133590 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 06 13:57:31.133811 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133797 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 06 13:57:31.133811 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133813 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 06 13:57:31.133962 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133819 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 06 13:57:31.133962 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133825 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 06 13:57:31.133962 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133835 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 06 13:57:31.133962 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.133843 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 06 13:57:31.134493 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.134481 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 06 13:57:31.134493 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.134493 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 06 13:57:31.134554 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.134508 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 06 13:57:31.134554 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.134514 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 06 13:57:31.134554 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.134522 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 06 13:57:31.134554 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.134530 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 06 13:57:31.136156 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.136142 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 06 13:57:31.136248 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.136158 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 06 13:57:31.137338 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.137276 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 06 13:57:31.137338 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.137296 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 06 13:57:31.139636 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.139470 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 06 13:57:31.141196 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.141178 2550 server.go:1268] "Started kubelet" Sep 06 13:57:31.141312 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.141266 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 06 13:57:31.141388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.141328 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 06 13:57:31.141496 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.141411 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 06 13:57:31.142065 ip-10-0-141-9 systemd[1]: Started Kubernetes Kubelet. Sep 06 13:57:31.145343 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.143340 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 06 13:57:31.145343 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.144436 2550 server.go:310] "Adding debug handlers to kubelet server" Sep 06 13:57:31.146150 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.146123 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 06 13:57:31.148721 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.148703 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 06 13:57:31.148721 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.148710 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 06 13:57:31.148856 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.148773 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.149227 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.148037 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-9.ec2.internal.18d2bff2b3ecced0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-9.ec2.internal,UID:ip-10-0-141-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-9.ec2.internal,},FirstTimestamp:2026-09-06 13:57:31.141140176 +0000 UTC m=+0.404584891,LastTimestamp:2026-09-06 13:57:31.141140176 +0000 UTC m=+0.404584891,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-9.ec2.internal,}" Sep 06 13:57:31.149466 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.149441 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 06 13:57:31.149547 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.149533 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 06 13:57:31.149604 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.149553 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 06 13:57:31.149692 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.149679 2550 reconstruct.go:97] "Volume reconstruction finished" Sep 06 13:57:31.149692 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.149692 2550 reconciler.go:29] "Reconciler: start to sync state" Sep 06 13:57:31.150030 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150016 2550 factory.go:55] Registering systemd factory Sep 06 13:57:31.150125 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150033 2550 factory.go:223] Registration of the systemd container factory successfully Sep 06 13:57:31.150304 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150277 2550 factory.go:153] Registering CRI-O factory Sep 06 13:57:31.150304 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150292 2550 factory.go:223] Registration of the crio container factory successfully Sep 06 13:57:31.150395 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150340 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 06 13:57:31.150395 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150359 2550 factory.go:103] Registering Raw factory Sep 06 13:57:31.150395 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150374 2550 manager.go:1196] Started watching for new ooms in manager Sep 06 13:57:31.150786 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.150772 2550 manager.go:349] Starting recovery of all containers Sep 06 13:57:31.157664 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.157627 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 06 13:57:31.157839 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.157736 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 06 13:57:31.160834 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.160797 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jhdt5" Sep 06 13:57:31.161059 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.161042 2550 manager.go:354] Recovery completed Sep 06 13:57:31.164191 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.164161 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jhdt5" Sep 06 13:57:31.167663 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.167646 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.169644 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.169622 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.169719 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.169662 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.169719 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.169678 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.170307 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.170289 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 06 13:57:31.170307 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.170304 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 06 13:57:31.170307 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.170307 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.170472 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.170320 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 06 13:57:31.171084 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.171068 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.171136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.171098 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.171136 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.171112 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.171332 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.171255 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-9.ec2.internal.18d2bff2b59fbdf9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-9.ec2.internal,UID:ip-10-0-141-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-9.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-9.ec2.internal,},FirstTimestamp:2026-09-06 13:57:31.169644025 +0000 UTC m=+0.433088741,LastTimestamp:2026-09-06 13:57:31.169644025 +0000 UTC m=+0.433088741,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-9.ec2.internal,}" Sep 06 13:57:31.172371 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.172358 2550 policy_none.go:49] "None policy: Start" Sep 06 13:57:31.172425 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.172374 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 06 13:57:31.172425 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.172385 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 06 13:57:31.174403 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.174385 2550 policy_none.go:47] "Start" Sep 06 13:57:31.218709 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.218688 2550 manager.go:337] "Starting Device Plugin manager" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.218751 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.218766 2550 server.go:87] "Starting device plugin registration server" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.219233 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.219250 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.219359 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.219456 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.219466 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.220107 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.220170 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.227418 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.228701 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.228753 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.228800 2550 kubelet.go:2448] "Starting kubelet main sync loop" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.228881 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 06 13:57:31.235820 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.230991 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:31.319797 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.319695 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.320765 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.320736 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.320765 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.320768 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.320955 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.320779 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.320955 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.320806 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.329294 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.329256 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"] Sep 06 13:57:31.329392 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.329346 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.330292 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.330274 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.330378 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.330307 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.330378 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.330320 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.330509 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.330495 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.330543 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.330518 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-9.ec2.internal\": node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.332606 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.332591 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.332733 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.332714 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.332792 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.332761 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.333478 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.333460 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.333564 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.333492 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.333564 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.333504 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.333564 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.333464 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.333650 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.333572 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.333650 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.333584 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.335880 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.335861 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.335994 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.335907 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 13:57:31.336675 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.336659 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory" Sep 06 13:57:31.336745 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.336685 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 13:57:31.336745 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.336694 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID" Sep 06 13:57:31.351502 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.351464 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e79bb2805f50e77c363fc54fc65fe32-config\") pod \"kube-apiserver-proxy-ip-10-0-141-9.ec2.internal\" (UID: \"4e79bb2805f50e77c363fc54fc65fe32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.351502 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.351499 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83631e085fe0c12b5412351d1accf045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"83631e085fe0c12b5412351d1accf045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.351701 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.351522 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83631e085fe0c12b5412351d1accf045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"83631e085fe0c12b5412351d1accf045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.365953 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.365925 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-9.ec2.internal\" not found" node="ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.370549 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.370528 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-9.ec2.internal\" not found" node="ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.374805 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.374783 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.451688 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.451653 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83631e085fe0c12b5412351d1accf045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"83631e085fe0c12b5412351d1accf045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.451688 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.451689 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83631e085fe0c12b5412351d1accf045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"83631e085fe0c12b5412351d1accf045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.451930 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.451707 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e79bb2805f50e77c363fc54fc65fe32-config\") pod \"kube-apiserver-proxy-ip-10-0-141-9.ec2.internal\" (UID: \"4e79bb2805f50e77c363fc54fc65fe32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.451930 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.451741 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83631e085fe0c12b5412351d1accf045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"83631e085fe0c12b5412351d1accf045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.451930 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.451755 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83631e085fe0c12b5412351d1accf045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"83631e085fe0c12b5412351d1accf045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.451930 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.451782 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e79bb2805f50e77c363fc54fc65fe32-config\") pod \"kube-apiserver-proxy-ip-10-0-141-9.ec2.internal\" (UID: \"4e79bb2805f50e77c363fc54fc65fe32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.474981 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.474940 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.575592 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.575516 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.670176 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.670137 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.674330 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:31.674309 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" Sep 06 13:57:31.676870 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.676604 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.778207 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.778162 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.879326 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.879245 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:31.980325 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:31.980282 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:32.082467 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:32.082416 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:32.082467 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.082456 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 06 13:57:32.083046 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.082614 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 13:57:32.168003 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.167962 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 13:52:32 +0000 UTC" deadline="2028-06-03 06:03:19.19090717 +0000 UTC" Sep 06 13:57:32.168003 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.168000 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15256h5m47.022911057s" Sep 06 13:57:32.183567 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:32.183529 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:32.203226 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.203196 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:32.285419 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:32.285384 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:32.386238 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:32.386197 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:32.451918 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.451802 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:32.487088 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:32.487042 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found" Sep 06 13:57:32.555557 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:32.555513 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e79bb2805f50e77c363fc54fc65fe32.slice/crio-db8ff06e7d6dfbead0fd7e5fdcb1beb71239547c10388e521c85cb406de5d79a WatchSource:0}: Error finding container db8ff06e7d6dfbead0fd7e5fdcb1beb71239547c10388e521c85cb406de5d79a: Status 404 returned error can't find the container with id db8ff06e7d6dfbead0fd7e5fdcb1beb71239547c10388e521c85cb406de5d79a Sep 06 13:57:32.556154 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:32.556133 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83631e085fe0c12b5412351d1accf045.slice/crio-ecb5b1603dcdfa0fe60e5e57fbc7acfc27d9a36b56e4663bd15caddebd45ed65 WatchSource:0}: Error finding container ecb5b1603dcdfa0fe60e5e57fbc7acfc27d9a36b56e4663bd15caddebd45ed65: Status 404 returned error can't find the container with id ecb5b1603dcdfa0fe60e5e57fbc7acfc27d9a36b56e4663bd15caddebd45ed65 Sep 06 13:57:32.562253 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.562231 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 13:57:32.565188 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.565166 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:32.648909 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.648864 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" Sep 06 13:57:32.663010 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.662968 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 13:57:32.664216 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.664196 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" Sep 06 13:57:32.672041 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:32.672016 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 13:57:33.005085 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.005052 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 13:57:33.128749 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.128686 2550 apiserver.go:52] "Watching apiserver" Sep 06 13:57:33.136383 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.136351 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 06 13:57:33.137721 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.137651 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk","openshift-multus/network-metrics-daemon-tw57c","openshift-network-diagnostics/network-check-target-bxwfx","openshift-network-operator/iptables-alerter-zsxrz","openshift-ovn-kubernetes/ovnkube-node-wz8jk","kube-system/global-pull-secret-syncer-b7m9q","kube-system/konnectivity-agent-zh6zz","openshift-cluster-node-tuning-operator/tuned-2bqjv","openshift-dns/node-resolver-8hv2l","openshift-image-registry/node-ca-nq79q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal","openshift-multus/multus-additional-cni-plugins-lzz6f","openshift-multus/multus-xkbv8"] Sep 06 13:57:33.140437 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.140409 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.140564 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.140521 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:33.142996 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.142969 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.145541 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.145508 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.145663 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.145614 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:33.145877 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.145856 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 06 13:57:33.146004 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.145974 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.146101 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.145974 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 06 13:57:33.146208 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.146191 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 06 13:57:33.146516 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.146392 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fnwx9\"" Sep 06 13:57:33.146516 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.146426 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.148094 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.148069 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:33.148214 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.148139 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:33.150315 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.150292 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 06 13:57:33.152479 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.152454 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.154812 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.154783 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bzfm2\"" Sep 06 13:57:33.155127 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.155107 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.155325 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.155298 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.155441 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.155326 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 06 13:57:33.155503 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.155483 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.158026 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.158003 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.158520 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.158230 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.159009 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.158900 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 06 13:57:33.159497 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.159230 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.159497 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.159264 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 06 13:57:33.159497 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.159481 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w2tnm\"" Sep 06 13:57:33.159776 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.159759 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 06 13:57:33.159859 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.159765 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 06 13:57:33.160632 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.160352 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c61bfe89-af85-4a64-a403-860883b0890b-dbus\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.162213 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.161851 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.162213 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.162146 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fg5mg\"" Sep 06 13:57:33.162859 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.162839 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.163455 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163062 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.163455 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163167 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-system-cni-dir\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.163455 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163314 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.163455 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163438 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rh7\" (UniqueName: \"kubernetes.io/projected/bec424d7-8783-4afa-9fe8-779b76b6b015-kube-api-access-74rh7\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.163732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163483 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c61bfe89-af85-4a64-a403-860883b0890b-kubelet-config\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.163732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-cnibin\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.163732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163540 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.163732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:33.163732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163597 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.163732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163729 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bec424d7-8783-4afa-9fe8-779b76b6b015-iptables-alerter-script\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163774 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163805 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdjp\" (UniqueName: \"kubernetes.io/projected/7025a487-10f2-475a-8da2-30f50c9989c0-kube-api-access-4gdjp\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bec424d7-8783-4afa-9fe8-779b76b6b015-host-slash\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163867 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163890 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.164056 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.163944 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7btx\" (UniqueName: \"kubernetes.io/projected/07010b81-f41e-4dd2-aced-6ce550d937ce-kube-api-access-q7btx\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.166659 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.166635 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 06 13:57:33.167380 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.167356 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ml9b7\"" Sep 06 13:57:33.168780 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.168296 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.171036 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.170742 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.171036 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.170898 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.171174 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.171145 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.171547 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.171322 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 06 13:57:33.171547 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.171366 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-drm4s\"" Sep 06 13:57:33.173481 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.173308 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.173481 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.173392 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cxmpn\"" Sep 06 13:57:33.173481 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.173411 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.174060 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.173889 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.176181 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.176162 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 06 13:57:33.176415 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.176398 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.176679 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.176660 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbflc\"" Sep 06 13:57:33.176679 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.176672 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 06 13:57:33.176809 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.176668 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 06 13:57:33.178678 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.178660 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 06 13:57:33.178825 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.178803 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 06 13:57:33.179083 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.179064 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v6h5m\"" Sep 06 13:57:33.186587 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.186561 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2np87" Sep 06 13:57:33.193016 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.192988 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2np87" Sep 06 13:57:33.236059 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.236002 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" event={"ID":"83631e085fe0c12b5412351d1accf045","Type":"ContainerStarted","Data":"ecb5b1603dcdfa0fe60e5e57fbc7acfc27d9a36b56e4663bd15caddebd45ed65"} Sep 06 13:57:33.238303 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.238274 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" event={"ID":"4e79bb2805f50e77c363fc54fc65fe32","Type":"ContainerStarted","Data":"db8ff06e7d6dfbead0fd7e5fdcb1beb71239547c10388e521c85cb406de5d79a"} Sep 06 13:57:33.254914 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.253463 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 06 13:57:33.264391 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-var-lib-kubelet\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.264391 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264346 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ht4s\" (UniqueName: \"kubernetes.io/projected/b3ff0bbe-649f-4370-9877-d4fe5b184c99-kube-api-access-2ht4s\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.264391 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264382 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264408 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-env-overrides\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264434 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ql7\" (UniqueName: \"kubernetes.io/projected/94411733-7637-47ae-a05c-30a723c94748-kube-api-access-q2ql7\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264459 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-lib-modules\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264483 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-tuned\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264509 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c61bfe89-af85-4a64-a403-860883b0890b-dbus\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264547 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-system-cni-dir\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264573 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-cni-bin\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264612 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ed959d9-887f-4d39-9f00-015d657dd5ae-konnectivity-ca\") pod \"konnectivity-agent-zh6zz\" (UID: \"6ed959d9-887f-4d39-9f00-015d657dd5ae\") " pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264638 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-host\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264662 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-netns\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.264686 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264687 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-var-lib-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264714 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c61bfe89-af85-4a64-a403-860883b0890b-kubelet-config\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264739 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-socket-dir-parent\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264773 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264799 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-systemd\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264831 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-systemd\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264863 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-sys\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264888 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-cni-multus\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264915 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7btx\" (UniqueName: \"kubernetes.io/projected/07010b81-f41e-4dd2-aced-6ce550d937ce-kube-api-access-q7btx\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264941 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-etc-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.264985 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-log-socket\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265010 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-kubernetes\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265034 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-cnibin\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265059 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1e2bb6-a11c-4520-af37-b832772ecd1b-tmp-dir\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265085 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265110 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-kubelet\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-cni-netd\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.265218 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265159 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-socket-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265182 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265203 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqjf\" (UniqueName: \"kubernetes.io/projected/1a1e2bb6-a11c-4520-af37-b832772ecd1b-kube-api-access-9zqjf\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265256 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ed959d9-887f-4d39-9f00-015d657dd5ae-agent-certs\") pod \"konnectivity-agent-zh6zz\" (UID: \"6ed959d9-887f-4d39-9f00-015d657dd5ae\") " pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.265294 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265380 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c61bfe89-af85-4a64-a403-860883b0890b-dbus\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.265394 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:33.765367262 +0000 UTC m=+3.028811979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265395 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-system-cni-dir\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265297 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysconfig\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265399 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c61bfe89-af85-4a64-a403-860883b0890b-kubelet-config\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265460 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysctl-conf\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265496 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265525 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-slash\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265547 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovn-node-metrics-cert\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265583 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-system-cni-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.266103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265612 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-multus-certs\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265643 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdjp\" (UniqueName: \"kubernetes.io/projected/7025a487-10f2-475a-8da2-30f50c9989c0-kube-api-access-4gdjp\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265677 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bec424d7-8783-4afa-9fe8-779b76b6b015-host-slash\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265732 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-device-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265825 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bec424d7-8783-4afa-9fe8-779b76b6b015-host-slash\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265831 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysctl-d\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265880 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3ff0bbe-649f-4370-9877-d4fe5b184c99-cni-binary-copy\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265928 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovnkube-config\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265970 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.265996 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-modprobe-d\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-k8s-cni-cncf-io\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266049 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266088 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-registration-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266125 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266131 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94411733-7637-47ae-a05c-30a723c94748-host\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.267024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266186 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266223 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266328 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266378 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovnkube-script-lib\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266408 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-sys-fs\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266435 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4b7k\" (UniqueName: \"kubernetes.io/projected/c6c69a5f-0941-475d-801f-5ebc9578a160-kube-api-access-s4b7k\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266440 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266462 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-run\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266488 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-cni-bin\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-hostroot\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266584 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a1e2bb6-a11c-4520-af37-b832772ecd1b-hosts-file\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266622 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tffjz\" (UniqueName: \"kubernetes.io/projected/a58e84ca-acb0-4398-9931-c351d62140ab-kube-api-access-tffjz\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266684 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-ovn\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266716 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266744 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rh7\" (UniqueName: \"kubernetes.io/projected/bec424d7-8783-4afa-9fe8-779b76b6b015-kube-api-access-74rh7\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266769 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-run-netns\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.267832 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.266812 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.266883 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:57:33.766863329 +0000 UTC m=+3.030308054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266881 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07010b81-f41e-4dd2-aced-6ce550d937ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266925 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zjf\" (UniqueName: \"kubernetes.io/projected/cd9b5231-d52d-42af-9806-8a47b749f4d7-kube-api-access-q9zjf\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266964 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-cni-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.266992 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-kubelet\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267017 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94411733-7637-47ae-a05c-30a723c94748-serviceca\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-cnibin\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267116 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bec424d7-8783-4afa-9fe8-779b76b6b015-iptables-alerter-script\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267146 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-node-log\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267151 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07010b81-f41e-4dd2-aced-6ce550d937ce-cnibin\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267169 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6c69a5f-0941-475d-801f-5ebc9578a160-tmp\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267198 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-conf-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267225 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-daemon-config\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267251 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-etc-kubernetes\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267276 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-systemd-units\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.268649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.267751 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bec424d7-8783-4afa-9fe8-779b76b6b015-iptables-alerter-script\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.272866 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.272839 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:33.272866 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.272870 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:33.273040 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.272884 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:33.273040 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.272988 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:33.772956104 +0000 UTC m=+3.036400813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:33.274028 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.273996 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 06 13:57:33.279307 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.279148 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rh7\" (UniqueName: \"kubernetes.io/projected/bec424d7-8783-4afa-9fe8-779b76b6b015-kube-api-access-74rh7\") pod \"iptables-alerter-zsxrz\" (UID: \"bec424d7-8783-4afa-9fe8-779b76b6b015\") " pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.279454 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.279150 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdjp\" (UniqueName: \"kubernetes.io/projected/7025a487-10f2-475a-8da2-30f50c9989c0-kube-api-access-4gdjp\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.279454 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.279225 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7btx\" (UniqueName: \"kubernetes.io/projected/07010b81-f41e-4dd2-aced-6ce550d937ce-kube-api-access-q7btx\") pod \"multus-additional-cni-plugins-lzz6f\" (UID: \"07010b81-f41e-4dd2-aced-6ce550d937ce\") " pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.368358 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368318 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovnkube-config\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.368549 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368372 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.368549 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368403 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-modprobe-d\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.368549 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368432 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-k8s-cni-cncf-io\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.368549 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368458 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.368549 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368487 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-registration-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.368549 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368512 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94411733-7637-47ae-a05c-30a723c94748-host\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368560 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368589 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368617 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovnkube-script-lib\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368642 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-sys-fs\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368668 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4b7k\" (UniqueName: \"kubernetes.io/projected/c6c69a5f-0941-475d-801f-5ebc9578a160-kube-api-access-s4b7k\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368692 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-run\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368717 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-cni-bin\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368748 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-hostroot\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368785 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a1e2bb6-a11c-4520-af37-b832772ecd1b-hosts-file\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368810 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tffjz\" (UniqueName: \"kubernetes.io/projected/a58e84ca-acb0-4398-9931-c351d62140ab-kube-api-access-tffjz\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368840 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-ovn\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.368890 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368880 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-run-netns\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368905 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zjf\" (UniqueName: \"kubernetes.io/projected/cd9b5231-d52d-42af-9806-8a47b749f4d7-kube-api-access-q9zjf\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368932 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-cni-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368971 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-kubelet\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.368997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94411733-7637-47ae-a05c-30a723c94748-serviceca\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369023 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-node-log\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369050 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6c69a5f-0941-475d-801f-5ebc9578a160-tmp\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-conf-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369113 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-daemon-config\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369137 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-etc-kubernetes\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369166 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-systemd-units\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369191 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-var-lib-kubelet\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369219 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ht4s\" (UniqueName: \"kubernetes.io/projected/b3ff0bbe-649f-4370-9877-d4fe5b184c99-kube-api-access-2ht4s\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369248 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-env-overrides\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369280 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ql7\" (UniqueName: \"kubernetes.io/projected/94411733-7637-47ae-a05c-30a723c94748-kube-api-access-q2ql7\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369311 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-lib-modules\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369336 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-tuned\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369363 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-cni-bin\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.369528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369390 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ed959d9-887f-4d39-9f00-015d657dd5ae-konnectivity-ca\") pod \"konnectivity-agent-zh6zz\" (UID: \"6ed959d9-887f-4d39-9f00-015d657dd5ae\") " pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369414 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-host\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369440 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-netns\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369466 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-var-lib-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369495 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-socket-dir-parent\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369542 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-systemd\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369568 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-systemd\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369592 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-sys\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369630 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-cni-multus\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369659 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-etc-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369684 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-log-socket\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369711 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-kubernetes\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369736 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-cnibin\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369762 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1e2bb6-a11c-4520-af37-b832772ecd1b-tmp-dir\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369792 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-kubelet\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369818 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-cni-netd\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369845 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-socket-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369884 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.370457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369911 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqjf\" (UniqueName: \"kubernetes.io/projected/1a1e2bb6-a11c-4520-af37-b832772ecd1b-kube-api-access-9zqjf\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369942 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ed959d9-887f-4d39-9f00-015d657dd5ae-agent-certs\") pod \"konnectivity-agent-zh6zz\" (UID: \"6ed959d9-887f-4d39-9f00-015d657dd5ae\") " pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.369984 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysconfig\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370011 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysctl-conf\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370039 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-slash\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370066 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovn-node-metrics-cert\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-system-cni-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370132 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-multus-certs\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370165 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-device-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370205 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysctl-d\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370233 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3ff0bbe-649f-4370-9877-d4fe5b184c99-cni-binary-copy\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.371388 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.370992 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3ff0bbe-649f-4370-9877-d4fe5b184c99-cni-binary-copy\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371557 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovnkube-config\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371641 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-modprobe-d\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371794 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-k8s-cni-cncf-io\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371834 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371883 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-registration-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.371934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371925 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94411733-7637-47ae-a05c-30a723c94748-host\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.372304 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.371958 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.372304 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372031 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.372557 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372531 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovnkube-script-lib\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.372646 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372619 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-host\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.372705 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372685 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-netns\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.372756 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372733 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-var-lib-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.372875 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372853 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-socket-dir-parent\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.372980 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372903 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-systemd\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.372980 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.372956 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-systemd\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.373092 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373024 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-sys\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.373092 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373070 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-cni-multus\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.373092 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373079 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ed959d9-887f-4d39-9f00-015d657dd5ae-konnectivity-ca\") pod \"konnectivity-agent-zh6zz\" (UID: \"6ed959d9-887f-4d39-9f00-015d657dd5ae\") " pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.373263 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373122 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-etc-openvswitch\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.373263 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373166 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-log-socket\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.373263 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373245 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-kubernetes\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.373433 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373264 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-sys-fs\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.373433 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373297 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-cnibin\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.373584 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373565 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1e2bb6-a11c-4520-af37-b832772ecd1b-tmp-dir\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.373647 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373590 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-run\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.373647 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373638 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-cni-bin\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.373750 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373664 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-kubelet\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.373750 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373687 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-hostroot\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.373750 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373708 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-cni-netd\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.373920 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373749 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a1e2bb6-a11c-4520-af37-b832772ecd1b-hosts-file\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.373920 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373804 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-socket-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.373920 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.373903 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.374083 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374051 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-run-ovn\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.374137 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374100 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-run-netns\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.374390 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374369 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-cni-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.374452 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374430 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-var-lib-kubelet\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.374658 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374635 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-var-lib-kubelet\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.374755 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374703 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-node-log\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.374960 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.374938 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94411733-7637-47ae-a05c-30a723c94748-serviceca\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.375528 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysctl-conf\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.375548 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cd9b5231-d52d-42af-9806-8a47b749f4d7-env-overrides\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.375602 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysconfig\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376112 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-daemon-config\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376139 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-cni-bin\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376205 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-etc-kubernetes\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376214 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-system-cni-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376262 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-systemd-units\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376303 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-lib-modules\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376306 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-multus-conf-dir\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3ff0bbe-649f-4370-9877-d4fe5b184c99-host-run-multus-certs\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376369 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a58e84ca-acb0-4398-9931-c351d62140ab-device-dir\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376390 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd9b5231-d52d-42af-9806-8a47b749f4d7-host-slash\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.376523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.376470 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-sysctl-d\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.378928 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.378876 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6c69a5f-0941-475d-801f-5ebc9578a160-tmp\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.379675 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.379649 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd9b5231-d52d-42af-9806-8a47b749f4d7-ovn-node-metrics-cert\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.381446 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.380169 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6c69a5f-0941-475d-801f-5ebc9578a160-etc-tuned\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.385147 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.385084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ht4s\" (UniqueName: \"kubernetes.io/projected/b3ff0bbe-649f-4370-9877-d4fe5b184c99-kube-api-access-2ht4s\") pod \"multus-xkbv8\" (UID: \"b3ff0bbe-649f-4370-9877-d4fe5b184c99\") " pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.386175 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.386144 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqjf\" (UniqueName: \"kubernetes.io/projected/1a1e2bb6-a11c-4520-af37-b832772ecd1b-kube-api-access-9zqjf\") pod \"node-resolver-8hv2l\" (UID: \"1a1e2bb6-a11c-4520-af37-b832772ecd1b\") " pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.386745 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.386692 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tffjz\" (UniqueName: \"kubernetes.io/projected/a58e84ca-acb0-4398-9931-c351d62140ab-kube-api-access-tffjz\") pod \"aws-ebs-csi-driver-node-p4fdk\" (UID: \"a58e84ca-acb0-4398-9931-c351d62140ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.386745 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.386715 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4b7k\" (UniqueName: \"kubernetes.io/projected/c6c69a5f-0941-475d-801f-5ebc9578a160-kube-api-access-s4b7k\") pod \"tuned-2bqjv\" (UID: \"c6c69a5f-0941-475d-801f-5ebc9578a160\") " pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.388057 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.388030 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ql7\" (UniqueName: \"kubernetes.io/projected/94411733-7637-47ae-a05c-30a723c94748-kube-api-access-q2ql7\") pod \"node-ca-nq79q\" (UID: \"94411733-7637-47ae-a05c-30a723c94748\") " pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.389602 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.389575 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ed959d9-887f-4d39-9f00-015d657dd5ae-agent-certs\") pod \"konnectivity-agent-zh6zz\" (UID: \"6ed959d9-887f-4d39-9f00-015d657dd5ae\") " pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.389694 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.389669 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zjf\" (UniqueName: \"kubernetes.io/projected/cd9b5231-d52d-42af-9806-8a47b749f4d7-kube-api-access-q9zjf\") pod \"ovnkube-node-wz8jk\" (UID: \"cd9b5231-d52d-42af-9806-8a47b749f4d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.455021 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.454980 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" Sep 06 13:57:33.469656 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.469598 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07010b81_f41e_4dd2_aced_6ce550d937ce.slice/crio-068d882a7ce2f99d6b977d54c422d0197e2d3cd420b780e2663bd2499fb9e24f WatchSource:0}: Error finding container 068d882a7ce2f99d6b977d54c422d0197e2d3cd420b780e2663bd2499fb9e24f: Status 404 returned error can't find the container with id 068d882a7ce2f99d6b977d54c422d0197e2d3cd420b780e2663bd2499fb9e24f Sep 06 13:57:33.470166 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.469926 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zsxrz" Sep 06 13:57:33.480653 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.480606 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:33.481723 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.481660 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec424d7_8783_4afa_9fe8_779b76b6b015.slice/crio-5392ce73078dace15554e9b7969f0f82af31cee40e141db06eebfeaf08f3e170 WatchSource:0}: Error finding container 5392ce73078dace15554e9b7969f0f82af31cee40e141db06eebfeaf08f3e170: Status 404 returned error can't find the container with id 5392ce73078dace15554e9b7969f0f82af31cee40e141db06eebfeaf08f3e170 Sep 06 13:57:33.484656 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.484641 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" Sep 06 13:57:33.493093 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.493021 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xkbv8" Sep 06 13:57:33.493370 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.493337 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9b5231_d52d_42af_9806_8a47b749f4d7.slice/crio-3bc77cf22d908e60985fc3a499e15009c5732b420c188e08ce2efa4eeb684e97 WatchSource:0}: Error finding container 3bc77cf22d908e60985fc3a499e15009c5732b420c188e08ce2efa4eeb684e97: Status 404 returned error can't find the container with id 3bc77cf22d908e60985fc3a499e15009c5732b420c188e08ce2efa4eeb684e97 Sep 06 13:57:33.497714 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.497673 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c69a5f_0941_475d_801f_5ebc9578a160.slice/crio-cea9bd7570b0f02346df8af1c74848170de644161fc64054a7d8a242f0bddb6c WatchSource:0}: Error finding container cea9bd7570b0f02346df8af1c74848170de644161fc64054a7d8a242f0bddb6c: Status 404 returned error can't find the container with id cea9bd7570b0f02346df8af1c74848170de644161fc64054a7d8a242f0bddb6c Sep 06 13:57:33.500517 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.500489 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" Sep 06 13:57:33.509383 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.509088 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hv2l" Sep 06 13:57:33.509729 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.509690 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ff0bbe_649f_4370_9877_d4fe5b184c99.slice/crio-6e6ea24001462511ada965c08916b3422baa27ea2e41ceaa62bfbe9543054aca WatchSource:0}: Error finding container 6e6ea24001462511ada965c08916b3422baa27ea2e41ceaa62bfbe9543054aca: Status 404 returned error can't find the container with id 6e6ea24001462511ada965c08916b3422baa27ea2e41ceaa62bfbe9543054aca Sep 06 13:57:33.518216 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.517934 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58e84ca_acb0_4398_9931_c351d62140ab.slice/crio-ab52f1c3915373b0f71ca59b321874ed519ae40cb5da495bb4d81431611a929b WatchSource:0}: Error finding container ab52f1c3915373b0f71ca59b321874ed519ae40cb5da495bb4d81431611a929b: Status 404 returned error can't find the container with id ab52f1c3915373b0f71ca59b321874ed519ae40cb5da495bb4d81431611a929b Sep 06 13:57:33.519413 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.519199 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nq79q" Sep 06 13:57:33.526400 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.526125 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:33.547095 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:57:33.547052 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed959d9_887f_4d39_9f00_015d657dd5ae.slice/crio-ef91fcd4c4455ccb65d8044883a94bbe284c91dad663cc1badd99ec001acd7e1 WatchSource:0}: Error finding container ef91fcd4c4455ccb65d8044883a94bbe284c91dad663cc1badd99ec001acd7e1: Status 404 returned error can't find the container with id ef91fcd4c4455ccb65d8044883a94bbe284c91dad663cc1badd99ec001acd7e1 Sep 06 13:57:33.773457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.773356 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:33.773457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.773413 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:33.773460 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773563 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773585 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773560 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773647 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:34.773627795 +0000 UTC m=+4.037072518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773650 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773665 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:33.773699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773666 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:57:34.773655558 +0000 UTC m=+4.037100265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:33.774031 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:33.773721 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:34.77370428 +0000 UTC m=+4.037149003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:34.197520 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.197385 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 13:52:33 +0000 UTC" deadline="2028-03-07 16:58:50.041007907 +0000 UTC" Sep 06 13:57:34.197520 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.197452 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13155h1m15.843560383s" Sep 06 13:57:34.245472 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.245323 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hv2l" event={"ID":"1a1e2bb6-a11c-4520-af37-b832772ecd1b","Type":"ContainerStarted","Data":"fa2b33cc37c14c3a708d547f0a0d9b52bc0063c07ef217d141e613d703605d8b"} Sep 06 13:57:34.247657 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.247598 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" event={"ID":"a58e84ca-acb0-4398-9931-c351d62140ab","Type":"ContainerStarted","Data":"ab52f1c3915373b0f71ca59b321874ed519ae40cb5da495bb4d81431611a929b"} Sep 06 13:57:34.250026 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.249990 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xkbv8" event={"ID":"b3ff0bbe-649f-4370-9877-d4fe5b184c99","Type":"ContainerStarted","Data":"6e6ea24001462511ada965c08916b3422baa27ea2e41ceaa62bfbe9543054aca"} Sep 06 13:57:34.253754 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.253722 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zh6zz" event={"ID":"6ed959d9-887f-4d39-9f00-015d657dd5ae","Type":"ContainerStarted","Data":"ef91fcd4c4455ccb65d8044883a94bbe284c91dad663cc1badd99ec001acd7e1"} Sep 06 13:57:34.256661 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.256626 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nq79q" event={"ID":"94411733-7637-47ae-a05c-30a723c94748","Type":"ContainerStarted","Data":"1c4e58e7e89f848559e30d632985ed0ce65d11dfee228dedad637153f629ecbc"} Sep 06 13:57:34.258265 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.258201 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" event={"ID":"c6c69a5f-0941-475d-801f-5ebc9578a160","Type":"ContainerStarted","Data":"cea9bd7570b0f02346df8af1c74848170de644161fc64054a7d8a242f0bddb6c"} Sep 06 13:57:34.260490 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.260357 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"3bc77cf22d908e60985fc3a499e15009c5732b420c188e08ce2efa4eeb684e97"} Sep 06 13:57:34.263524 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.263456 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zsxrz" event={"ID":"bec424d7-8783-4afa-9fe8-779b76b6b015","Type":"ContainerStarted","Data":"5392ce73078dace15554e9b7969f0f82af31cee40e141db06eebfeaf08f3e170"} Sep 06 13:57:34.266687 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.266612 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerStarted","Data":"068d882a7ce2f99d6b977d54c422d0197e2d3cd420b780e2663bd2499fb9e24f"} Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.782252 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.782390 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:34.782452 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.782622 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.782695 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:36.782675377 +0000 UTC m=+6.046120099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.783201 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.783223 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.783238 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.783290 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:36.783273037 +0000 UTC m=+6.046717767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.783356 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:34.783431 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:34.783393 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:57:36.783381281 +0000 UTC m=+6.046826006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:35.198542 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:35.198455 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 13:52:33 +0000 UTC" deadline="2028-03-10 20:22:37.727420581 +0000 UTC" Sep 06 13:57:35.198542 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:35.198504 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13230h25m2.528921197s" Sep 06 13:57:35.232865 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:35.232025 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:35.232865 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:35.232153 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:35.232865 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:35.232485 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:35.232865 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:35.232560 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:35.232865 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:35.232626 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:35.232865 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:35.232689 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:36.803310 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:36.803267 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:36.803328 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:36.803381 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803480 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803514 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803531 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803569 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803597 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:40.803577447 +0000 UTC m=+10.067022173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803622 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:57:40.803606188 +0000 UTC m=+10.067050895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803513 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:36.804055 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:36.803657 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:40.803647727 +0000 UTC m=+10.067092430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:37.230679 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:37.229697 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:37.230679 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:37.229853 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:37.231118 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:37.231076 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:37.231355 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:37.231074 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:37.231355 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:37.231226 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:37.231355 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:37.231323 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:39.232220 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:39.232191 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:39.232594 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:39.232338 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:39.232785 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:39.232766 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:39.233060 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:39.232996 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:39.233152 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:39.233088 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:39.233228 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:39.233201 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:40.843337 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:40.843289 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:40.843365 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:40.843404 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843467 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843521 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843538 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843551 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:48.843528835 +0000 UTC m=+18.106973560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843560 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843570 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:57:48.843558574 +0000 UTC m=+18.107003277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843575 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:40.843995 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:40.843612 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:48.843601744 +0000 UTC m=+18.107046454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:41.235121 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:41.235068 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:41.235891 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:41.235430 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:41.235891 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:41.235566 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:41.235891 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:41.235598 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:41.235891 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:41.235725 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:41.235891 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:41.235852 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:43.230939 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:43.230076 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:43.230939 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:43.230237 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:43.230939 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:43.230696 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:43.230939 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:43.230801 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:43.231555 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:43.230937 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:43.231555 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:43.231039 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:44.295366 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:44.295305 2550 generic.go:356] "Generic (PLEG): container finished" podID="83631e085fe0c12b5412351d1accf045" containerID="bbda0bbfd7b415ed9c7c3b34afe9ed71ed0adfc0e2735f2e83fc584e8691bec6" exitCode=0 Sep 06 13:57:44.295366 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:44.295345 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" event={"ID":"83631e085fe0c12b5412351d1accf045","Type":"ContainerDied","Data":"bbda0bbfd7b415ed9c7c3b34afe9ed71ed0adfc0e2735f2e83fc584e8691bec6"} Sep 06 13:57:45.229341 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:45.229305 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:45.229341 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:45.229333 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:45.229603 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:45.229305 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:45.229603 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:45.229442 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:45.229603 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:45.229520 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:45.229768 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:45.229604 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:47.229237 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:47.229192 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:47.229712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:47.229192 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:47.229712 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:47.229347 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:47.229712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:47.229192 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:47.229712 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:47.229465 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:47.229712 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:47.229510 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:48.904279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:48.904189 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:48.904279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:48.904257 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:48.904289 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904348 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904371 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904419 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:58:04.904400238 +0000 UTC m=+34.167844954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904435 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:04.904426662 +0000 UTC m=+34.167871368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904458 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904487 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904527 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:48.904767 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:48.904589 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:04.904571803 +0000 UTC m=+34.168016518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:57:49.229792 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:49.229755 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:49.229997 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:49.229755 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:49.229997 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:49.229903 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:49.230108 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:49.229998 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:49.230108 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:49.229755 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:49.230200 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:49.230113 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:51.230324 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:51.230291 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:51.230779 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:51.230424 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:51.230779 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:51.230448 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:51.230779 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:51.230562 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:51.230779 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:51.230606 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:51.230779 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:51.230721 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:53.229832 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.229488 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:53.230475 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.229511 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:53.230475 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.229488 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:53.230475 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:53.229960 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:53.230475 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:53.230062 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:53.230475 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:53.230135 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:53.323835 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.323743 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" event={"ID":"83631e085fe0c12b5412351d1accf045","Type":"ContainerStarted","Data":"5290428edab23dcad1c00ac905e44982b383d5a5b5a3a5d696a4f1393444274c"} Sep 06 13:57:53.325486 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.325454 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zh6zz" event={"ID":"6ed959d9-887f-4d39-9f00-015d657dd5ae","Type":"ContainerStarted","Data":"4dd8ef6576624412bcaf3f8a0aa80d85903fe8827fb8bf312b025b88a5ea02a8"} Sep 06 13:57:53.327082 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.327055 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nq79q" event={"ID":"94411733-7637-47ae-a05c-30a723c94748","Type":"ContainerStarted","Data":"eef5334f2bfd594bc50e81d333188e4f76338da2f0b61ca6bf685caa82e70a56"} Sep 06 13:57:53.329178 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.329150 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" event={"ID":"c6c69a5f-0941-475d-801f-5ebc9578a160","Type":"ContainerStarted","Data":"cfdd41fe4e32fa929ca5e475b415b1440563825081d43ae1ae1d5aec2974a74b"} Sep 06 13:57:53.332746 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.332718 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"e20398f929ea0bfd07941308981a92bab4b6b86132ed40777631e38041ac05c1"} Sep 06 13:57:53.332869 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.332795 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"d0cfbdd34ebf9879e8742ea13c044d506cfa87ee55a335a8cda14178be6d4997"} Sep 06 13:57:53.332869 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.332844 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"ceb8e590f0c92793b99d80b0635d0c21458a66e08c5eefeed122c1a5a31998ad"} Sep 06 13:57:53.333016 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.332905 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"1ae295204d4f7e9e9cf0d9e6428fe69229d95d29e58ec0dbdb16fdcb79896a7f"} Sep 06 13:57:53.333016 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.332954 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"772e86c2b1cda9755b3986898a250c24e31858edec2067bf2a190241af9b6424"} Sep 06 13:57:53.333016 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.333000 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"31fc9814c0ceb8973f368503440d19e98f18c516ad29fe9f46b2652e4310c02d"} Sep 06 13:57:53.334523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.334494 2550 generic.go:356] "Generic (PLEG): container finished" podID="07010b81-f41e-4dd2-aced-6ce550d937ce" containerID="f249831e245bd14d9b472dd3eb78e7886fef493e7416aca56527d5d634b2cc4c" exitCode=0 Sep 06 13:57:53.334622 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.334574 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerDied","Data":"f249831e245bd14d9b472dd3eb78e7886fef493e7416aca56527d5d634b2cc4c"} Sep 06 13:57:53.336263 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.336240 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" event={"ID":"4e79bb2805f50e77c363fc54fc65fe32","Type":"ContainerStarted","Data":"f4ebf980fb366780af7004505c48565dad56ddf1e4a608a4e76061b9f1b3df99"} Sep 06 13:57:53.337794 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.337740 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" podStartSLOduration=20.337722785 podStartE2EDuration="20.337722785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.337295196 +0000 UTC m=+22.600739919" watchObservedRunningTime="2026-09-06 13:57:53.337722785 +0000 UTC m=+22.601167513" Sep 06 13:57:53.338661 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.338637 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hv2l" event={"ID":"1a1e2bb6-a11c-4520-af37-b832772ecd1b","Type":"ContainerStarted","Data":"e7753807a77363dd978a7d0b60bd11302d3cdcef953552e5099cdfb2f6d0102b"} Sep 06 13:57:53.340383 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.340362 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" event={"ID":"a58e84ca-acb0-4398-9931-c351d62140ab","Type":"ContainerStarted","Data":"e92369ffd91444eb983af77e32f149424e8c06bcf8a97671fda7452208e4df16"} Sep 06 13:57:53.342102 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.342068 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xkbv8" event={"ID":"b3ff0bbe-649f-4370-9877-d4fe5b184c99","Type":"ContainerStarted","Data":"6f3f15aea097287a8bbdc4fba59b6118238422364dc8926c4b7b050dc2dcfc8e"} Sep 06 13:57:53.352788 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.352729 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nq79q" podStartSLOduration=2.631410205 podStartE2EDuration="21.352713079s" totalImagesPullingTime="18.721302874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.35200018 +0000 UTC m=+22.615444905" watchObservedRunningTime="2026-09-06 13:57:53.352713079 +0000 UTC m=+22.616157802" Sep 06 13:57:53.368024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.367964 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zh6zz" podStartSLOduration=6.664045127 podStartE2EDuration="21.367944329s" totalImagesPullingTime="14.703899202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.367571071 +0000 UTC m=+22.631015796" watchObservedRunningTime="2026-09-06 13:57:53.367944329 +0000 UTC m=+22.631389053" Sep 06 13:57:53.380024 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.379963 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" podStartSLOduration=20.379940672 podStartE2EDuration="20.379940672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.379531758 +0000 UTC m=+22.642976482" watchObservedRunningTime="2026-09-06 13:57:53.379940672 +0000 UTC m=+22.643385412" Sep 06 13:57:53.412274 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.412217 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2bqjv" podStartSLOduration=2.651597748 podStartE2EDuration="21.412200718s" totalImagesPullingTime="18.76060297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.412088506 +0000 UTC m=+22.675533230" watchObservedRunningTime="2026-09-06 13:57:53.412200718 +0000 UTC m=+22.675645441" Sep 06 13:57:53.426211 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.426163 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8hv2l" podStartSLOduration=10.787246646 podStartE2EDuration="21.426147845s" totalImagesPullingTime="10.638901199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.425929499 +0000 UTC m=+22.689374220" watchObservedRunningTime="2026-09-06 13:57:53.426147845 +0000 UTC m=+22.689592569" Sep 06 13:57:53.577166 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:53.577009 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 06 13:57:54.232250 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.232069 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T13:57:53.577161425Z","UUID":"9d110c95-fd4f-4b09-be72-e147d69d153b","Handler":null,"Name":"","Endpoint":""} Sep 06 13:57:54.234286 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.233854 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 06 13:57:54.234286 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.233887 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 06 13:57:54.346458 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.346421 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zsxrz" event={"ID":"bec424d7-8783-4afa-9fe8-779b76b6b015","Type":"ContainerStarted","Data":"d2cf300252ec0f8fe9f6bafe200f3adfc26d0e34714738a92157c5ce97386dda"} Sep 06 13:57:54.349359 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.349321 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" event={"ID":"a58e84ca-acb0-4398-9931-c351d62140ab","Type":"ContainerStarted","Data":"8b4a9de2673e696e4537bfd290eceb68835dab960ec30d976bd9a14290c1e04f"} Sep 06 13:57:54.360009 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.359961 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xkbv8" podStartSLOduration=3.50431991 podStartE2EDuration="22.359939207s" totalImagesPullingTime="18.855619297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:53.447236387 +0000 UTC m=+22.710681112" watchObservedRunningTime="2026-09-06 13:57:54.359939207 +0000 UTC m=+23.623383931" Sep 06 13:57:54.360327 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:54.360292 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zsxrz" podStartSLOduration=3.582571542 podStartE2EDuration="22.360277954s" totalImagesPullingTime="18.777706412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:54.359674387 +0000 UTC m=+23.623119113" watchObservedRunningTime="2026-09-06 13:57:54.360277954 +0000 UTC m=+23.623722681" Sep 06 13:57:55.232697 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:55.232666 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:55.233196 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:55.232775 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:55.233196 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:55.232783 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:55.233196 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:55.232801 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:55.233196 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:55.232868 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:55.233196 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:55.232978 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:55.354859 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:55.354814 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"6cb61ce71b360ab44e674c53a55df72a1f674b431ad334650cb4f12cbb7259c1"} Sep 06 13:57:55.360197 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:55.360157 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" event={"ID":"a58e84ca-acb0-4398-9931-c351d62140ab","Type":"ContainerStarted","Data":"aef1a0773dd13d604d187beec5eaacccdffa72e19e50f067dd27c4bcbc834b8a"} Sep 06 13:57:55.375955 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:55.375883 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4fdk" podStartSLOduration=2.700892464 podStartE2EDuration="23.375862994s" totalImagesPullingTime="20.67497053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:55.375522945 +0000 UTC m=+24.638967670" watchObservedRunningTime="2026-09-06 13:57:55.375862994 +0000 UTC m=+24.639307719" Sep 06 13:57:56.556296 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:56.556259 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:56.557212 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:56.557193 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:57.229826 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:57.229748 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:57.230001 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:57.229748 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:57.230001 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:57.229748 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:57.230001 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:57.229957 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:57.230140 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:57.230011 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:57.230140 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:57.229918 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:57.792730 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:57.792469 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:57.793770 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:57.793300 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zh6zz" Sep 06 13:57:58.371408 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:58.371308 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" event={"ID":"cd9b5231-d52d-42af-9806-8a47b749f4d7","Type":"ContainerStarted","Data":"03eaf68317e043aed380988de93f127fd3392b6185696ce4873270b2f8099de3"} Sep 06 13:57:58.371634 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:58.371619 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:58.373284 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:58.373257 2550 generic.go:356] "Generic (PLEG): container finished" podID="07010b81-f41e-4dd2-aced-6ce550d937ce" containerID="c44697dcc63e164ec252071e2959288c63ddbdc424870394f903e93f2556896e" exitCode=0 Sep 06 13:57:58.376379 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:58.373298 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerDied","Data":"c44697dcc63e164ec252071e2959288c63ddbdc424870394f903e93f2556896e"} Sep 06 13:57:58.388631 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:58.388601 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:58.397207 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:58.397159 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" podStartSLOduration=7.6306123790000004 podStartE2EDuration="26.397143207s" totalImagesPullingTime="18.766530828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:58.396672705 +0000 UTC m=+27.660117430" watchObservedRunningTime="2026-09-06 13:57:58.397143207 +0000 UTC m=+27.660587983" Sep 06 13:57:59.229934 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.229886 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:59.230784 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.229886 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:59.230784 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.229886 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:59.230784 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:59.230020 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:59.230784 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:59.230156 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:57:59.230784 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:59.230233 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:59.379211 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.379171 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerStarted","Data":"8dd1635cc7fa15f452df4d3ab3fb096dbb524b11096992360530174d98b418dc"} Sep 06 13:57:59.380470 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.380226 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:59.380470 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.380309 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:59.395269 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.395242 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:57:59.618039 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.617952 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bxwfx"] Sep 06 13:57:59.618190 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.618136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:57:59.618251 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:59.618230 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:57:59.621396 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.621361 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7m9q"] Sep 06 13:57:59.621550 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.621469 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:57:59.621613 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:59.621559 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:57:59.622043 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.622011 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tw57c"] Sep 06 13:57:59.622150 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:57:59.622128 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:57:59.622259 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:57:59.622238 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:58:00.384202 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:00.384170 2550 generic.go:356] "Generic (PLEG): container finished" podID="07010b81-f41e-4dd2-aced-6ce550d937ce" containerID="8dd1635cc7fa15f452df4d3ab3fb096dbb524b11096992360530174d98b418dc" exitCode=0 Sep 06 13:58:00.384816 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:00.384260 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerDied","Data":"8dd1635cc7fa15f452df4d3ab3fb096dbb524b11096992360530174d98b418dc"} Sep 06 13:58:01.230043 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:01.229855 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:01.230186 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:01.229934 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:01.230186 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:01.230114 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:58:01.230290 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:01.229955 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:01.230290 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:01.230186 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:58:01.230290 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:01.230272 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:58:01.389224 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:01.389105 2550 generic.go:356] "Generic (PLEG): container finished" podID="07010b81-f41e-4dd2-aced-6ce550d937ce" containerID="e526e427f36910b5aa62e633d05197074ae1f96cc48f60628c5b095d7dda34b9" exitCode=0 Sep 06 13:58:01.389639 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:01.389203 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerDied","Data":"e526e427f36910b5aa62e633d05197074ae1f96cc48f60628c5b095d7dda34b9"} Sep 06 13:58:03.229560 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:03.229524 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:03.230057 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:03.229524 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:03.230057 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:03.229680 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tw57c" podUID="7025a487-10f2-475a-8da2-30f50c9989c0" Sep 06 13:58:03.230057 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:03.229753 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxwfx" podUID="00f7bb8f-55ef-4d41-ad99-af3d2ba7f185" Sep 06 13:58:03.230057 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:03.229524 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:03.230057 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:03.229847 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7m9q" podUID="c61bfe89-af85-4a64-a403-860883b0890b" Sep 06 13:58:04.920936 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:04.920874 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:04.920951 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:04.921006 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921070 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921086 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921100 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921114 2550 projected.go:196] Error preparing data for projected volume kube-api-access-vw4f9 for pod openshift-network-diagnostics/network-check-target-bxwfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921113 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921156 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret podName:c61bfe89-af85-4a64-a403-860883b0890b nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.921135956 +0000 UTC m=+66.184580677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret") pod "global-pull-secret-syncer-b7m9q" (UID: "c61bfe89-af85-4a64-a403-860883b0890b") : object "kube-system"/"original-pull-secret" not registered Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921175 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9 podName:00f7bb8f-55ef-4d41-ad99-af3d2ba7f185 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.92116602 +0000 UTC m=+66.184610724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vw4f9" (UniqueName: "kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9") pod "network-check-target-bxwfx" (UID: "00f7bb8f-55ef-4d41-ad99-af3d2ba7f185") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 13:58:04.921438 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:04.921194 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs podName:7025a487-10f2-475a-8da2-30f50c9989c0 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.921183761 +0000 UTC m=+66.184628463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs") pod "network-metrics-daemon-tw57c" (UID: "7025a487-10f2-475a-8da2-30f50c9989c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 13:58:05.038769 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.038745 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeReady" Sep 06 13:58:05.038950 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.038932 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 06 13:58:05.071352 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.071314 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7448bfcb7f-sjwzf"] Sep 06 13:58:05.096784 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.096685 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv"] Sep 06 13:58:05.098666 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.098044 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.100454 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.100428 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 06 13:58:05.100614 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.100496 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7kglp\"" Sep 06 13:58:05.100935 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.100910 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 06 13:58:05.102206 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.102168 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 06 13:58:05.106528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.105976 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 06 13:58:05.119949 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.119530 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c6c6bb47b-xc5lw"] Sep 06 13:58:05.120243 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.120224 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.122756 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.122724 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hzz5m\"" Sep 06 13:58:05.123164 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.123141 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.123269 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.123244 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.123365 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.123348 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 06 13:58:05.125423 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.125196 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 06 13:58:05.138160 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.138124 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vr22w"] Sep 06 13:58:05.138331 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.138266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.140536 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.140507 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 06 13:58:05.140774 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.140758 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 06 13:58:05.140955 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.140938 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2qrl5\"" Sep 06 13:58:05.141035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.140975 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.141114 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.141094 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.141189 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.141176 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 06 13:58:05.141298 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.141276 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 06 13:58:05.154524 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.154331 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf"] Sep 06 13:58:05.154748 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.154633 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.156816 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.156721 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zn6tl\"" Sep 06 13:58:05.156982 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.156847 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 06 13:58:05.156982 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.156869 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 06 13:58:05.157137 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.157115 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.157211 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.157153 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.161857 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.161798 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 06 13:58:05.172260 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.172218 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ss4gj"] Sep 06 13:58:05.172434 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.172360 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.175098 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.175071 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 06 13:58:05.175243 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.175146 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b29w2\"" Sep 06 13:58:05.175338 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.175318 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.175440 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.175422 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.190706 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.190661 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq"] Sep 06 13:58:05.190873 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.190785 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.192800 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.192777 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 06 13:58:05.193225 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.193206 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.193324 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.193262 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 06 13:58:05.193324 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.193262 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cf75l\"" Sep 06 13:58:05.193436 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.193369 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.197750 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.197719 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 06 13:58:05.204767 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.204739 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7448bfcb7f-sjwzf"] Sep 06 13:58:05.204767 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.204781 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pd4sj"] Sep 06 13:58:05.204993 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.204885 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.208103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.208077 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.208552 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.208529 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 06 13:58:05.209268 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.209244 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6vtcx\"" Sep 06 13:58:05.209541 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.209523 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 06 13:58:05.209678 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.209659 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.219028 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.219006 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-n82mx"] Sep 06 13:58:05.219173 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.219147 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.221909 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.221873 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 06 13:58:05.222019 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.221951 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mvgg4\"" Sep 06 13:58:05.222106 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.222085 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 06 13:58:05.223443 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223417 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.223553 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223460 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-installation-pull-secrets\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.223553 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223487 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-bound-sa-token\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.223553 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223520 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.223553 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223535 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.223553 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223551 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjtb\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-kube-api-access-rqjtb\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.223712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223579 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mjs\" (UniqueName: \"kubernetes.io/projected/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-kube-api-access-q2mjs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.223712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223612 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.223712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223630 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-trusted-ca\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.223712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223646 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.223712 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223678 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/019adef7-a1fb-4778-8880-2b206ee04a44-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.223852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223712 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.223852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223733 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-default-certificate\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.223852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223750 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-snapshots\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.223852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223772 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9gw\" (UniqueName: \"kubernetes.io/projected/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-kube-api-access-nr9gw\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.223852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223811 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-stats-auth\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.223852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223829 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-serving-cert\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.224093 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223854 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-image-registry-private-configuration\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.224093 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223876 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-certificates\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.224093 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.223976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-tmp\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.224093 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.224082 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0958c93-9d63-42d6-b109-ec5c7ae47828-ca-trust-extracted\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.224226 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.224105 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8dj\" (UniqueName: \"kubernetes.io/projected/019adef7-a1fb-4778-8880-2b206ee04a44-kube-api-access-vd8dj\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.239649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.239616 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:05.239830 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.239616 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:05.240016 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.239626 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:05.240163 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.239624 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" Sep 06 13:58:05.241824 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.241799 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htlhn\"" Sep 06 13:58:05.241994 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.241957 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cxw7\"" Sep 06 13:58:05.242740 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.242241 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.242740 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.242305 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 13:58:05.242740 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.242396 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.242740 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.242522 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 13:58:05.242740 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.242595 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w9c2g\"" Sep 06 13:58:05.244370 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.244351 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh"] Sep 06 13:58:05.260096 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.260012 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52"] Sep 06 13:58:05.260286 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.260266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.262528 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.262457 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 06 13:58:05.262624 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.262530 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4lsdn\"" Sep 06 13:58:05.262770 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.262753 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.262983 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.262963 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 06 13:58:05.263103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.263076 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.276087 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276065 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq"] Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276096 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv"] Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276118 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf"] Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276142 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ss4gj"] Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276161 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c6c6bb47b-xc5lw"] Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276178 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-n82mx"] Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276186 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" Sep 06 13:58:05.276198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276197 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh"] Sep 06 13:58:05.276535 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276214 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52"] Sep 06 13:58:05.276535 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.276234 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s6hq6"] Sep 06 13:58:05.278348 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.278328 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vpfsr\"" Sep 06 13:58:05.278483 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.278400 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.278483 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.278441 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.292575 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.292547 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pd4sj"] Sep 06 13:58:05.292752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.292593 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vr22w"] Sep 06 13:58:05.292752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.292624 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6hq6"] Sep 06 13:58:05.292948 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.292756 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.295076 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.295037 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 06 13:58:05.295192 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.295037 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dt8p5\"" Sep 06 13:58:05.295192 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.295043 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 06 13:58:05.295192 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.295086 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 06 13:58:05.324443 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324407 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-image-registry-private-configuration\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.324443 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324443 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-certificates\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324502 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be59bd6-458a-47f0-9e1f-4e22c97a8553-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324543 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324566 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjtb\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-kube-api-access-rqjtb\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324623 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dea50e1-1e6c-4f47-960c-6a53737fb691-config\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.324661 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 13:58:05.324685 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.324676 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7448bfcb7f-sjwzf: secret "image-registry-tls" not found Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324669 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.324778 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls podName:d0958c93-9d63-42d6-b109-ec5c7ae47828 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.824757277 +0000 UTC m=+35.088201995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls") pod "image-registry-7448bfcb7f-sjwzf" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828") : secret "image-registry-tls" not found Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324841 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e557111-94a1-498c-a4f9-273705d374be-trusted-ca\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324871 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/019adef7-a1fb-4778-8880-2b206ee04a44-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324914 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4s6t\" (UniqueName: \"kubernetes.io/projected/5f7af3e7-4253-4adc-9187-05a252402917-kube-api-access-q4s6t\") pod \"network-check-source-685456655f-n82mx\" (UID: \"5f7af3e7-4253-4adc-9187-05a252402917\") " pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324967 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-default-certificate\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.324991 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-snapshots\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325019 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9gw\" (UniqueName: \"kubernetes.io/projected/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-kube-api-access-nr9gw\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325044 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e557111-94a1-498c-a4f9-273705d374be-serving-cert\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325079 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-certificates\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325084 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4r4\" (UniqueName: \"kubernetes.io/projected/7be59bd6-458a-47f0-9e1f-4e22c97a8553-kube-api-access-mz4r4\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.325117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-serving-cert\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325131 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-tmp\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325154 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-installation-pull-secrets\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325175 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-bound-sa-token\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325214 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b13bd22-de6a-4432-92d0-8990aee7c811-tmp-dir\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325241 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k2d\" (UniqueName: \"kubernetes.io/projected/7dea50e1-1e6c-4f47-960c-6a53737fb691-kube-api-access-d9k2d\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0958c93-9d63-42d6-b109-ec5c7ae47828-ca-trust-extracted\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325285 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8dj\" (UniqueName: \"kubernetes.io/projected/019adef7-a1fb-4778-8880-2b206ee04a44-kube-api-access-vd8dj\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325310 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dea50e1-1e6c-4f47-960c-6a53737fb691-serving-cert\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325336 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvkc\" (UniqueName: \"kubernetes.io/projected/0a63c936-81b4-428d-b609-83a70a3e2ac6-kube-api-access-9tvkc\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325363 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325375 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325391 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mjs\" (UniqueName: \"kubernetes.io/projected/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-kube-api-access-q2mjs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325463 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6c9l\" (UniqueName: \"kubernetes.io/projected/0b13bd22-de6a-4432-92d0-8990aee7c811-kube-api-access-g6c9l\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325512 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-trusted-ca\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.325698 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325539 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.325665 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.825619083 +0000 UTC m=+35.089063812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : configmap references non-existent config key: service-ca.crt Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325726 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-snapshots\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325753 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/019adef7-a1fb-4778-8880-2b206ee04a44-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.325936 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e557111-94a1-498c-a4f9-273705d374be-config\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326072 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13bd22-de6a-4432-92d0-8990aee7c811-config-volume\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.326095 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326119 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95cv\" (UniqueName: \"kubernetes.io/projected/0e557111-94a1-498c-a4f9-273705d374be-kube-api-access-b95cv\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.326140 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.826125723 +0000 UTC m=+35.089570442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : secret "router-metrics-certs-default" not found Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326202 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326252 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-stats-auth\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326277 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7be59bd6-458a-47f0-9e1f-4e22c97a8553-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.326333 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326286 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.326824 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326358 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-tmp\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.326824 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.326370 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:05.326824 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.326480 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls podName:019adef7-a1fb-4778-8880-2b206ee04a44 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.826464683 +0000 UTC m=+35.089909402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gscwv" (UID: "019adef7-a1fb-4778-8880-2b206ee04a44") : secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:05.326824 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326610 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0958c93-9d63-42d6-b109-ec5c7ae47828-ca-trust-extracted\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.326824 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.326695 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-trusted-ca\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.329843 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.329821 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-serving-cert\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.330127 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.330088 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-default-certificate\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.330245 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.330134 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-stats-auth\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.330493 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.330472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-installation-pull-secrets\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.330611 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.330591 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-image-registry-private-configuration\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.333115 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.333068 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjtb\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-kube-api-access-rqjtb\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.333746 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.333705 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-bound-sa-token\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.334688 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.334653 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9gw\" (UniqueName: \"kubernetes.io/projected/b1209f84-910d-4d92-ae80-c5eaac7cdb1f-kube-api-access-nr9gw\") pod \"insights-operator-76bcd6ccf5-vr22w\" (UID: \"b1209f84-910d-4d92-ae80-c5eaac7cdb1f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.334788 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.334655 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8dj\" (UniqueName: \"kubernetes.io/projected/019adef7-a1fb-4778-8880-2b206ee04a44-kube-api-access-vd8dj\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.335085 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.335064 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mjs\" (UniqueName: \"kubernetes.io/projected/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-kube-api-access-q2mjs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.427286 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6c9l\" (UniqueName: \"kubernetes.io/projected/0b13bd22-de6a-4432-92d0-8990aee7c811-kube-api-access-g6c9l\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.427286 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427290 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qj4z\" (UniqueName: \"kubernetes.io/projected/46f7ff31-5a17-4585-b2f9-a2d4981f00d4-kube-api-access-7qj4z\") pod \"volume-data-source-validator-695c895bd6-b6w52\" (UID: \"46f7ff31-5a17-4585-b2f9-a2d4981f00d4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" Sep 06 13:58:05.427556 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427416 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce47be1d-02c9-4e8b-866b-2531c424bc2c-cert\") pod \"ingress-canary-s6hq6\" (UID: \"ce47be1d-02c9-4e8b-866b-2531c424bc2c\") " pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.427556 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427481 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e557111-94a1-498c-a4f9-273705d374be-config\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.427556 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427506 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13bd22-de6a-4432-92d0-8990aee7c811-config-volume\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.427556 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427531 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95cv\" (UniqueName: \"kubernetes.io/projected/0e557111-94a1-498c-a4f9-273705d374be-kube-api-access-b95cv\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.427752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427589 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7be59bd6-458a-47f0-9e1f-4e22c97a8553-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.427752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427618 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.427752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be59bd6-458a-47f0-9e1f-4e22c97a8553-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.427752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427695 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jqq\" (UniqueName: \"kubernetes.io/projected/ce47be1d-02c9-4e8b-866b-2531c424bc2c-kube-api-access-f8jqq\") pod \"ingress-canary-s6hq6\" (UID: \"ce47be1d-02c9-4e8b-866b-2531c424bc2c\") " pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.427752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dea50e1-1e6c-4f47-960c-6a53737fb691-config\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.428035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427759 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.428035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427788 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e557111-94a1-498c-a4f9-273705d374be-trusted-ca\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.428035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4s6t\" (UniqueName: \"kubernetes.io/projected/5f7af3e7-4253-4adc-9187-05a252402917-kube-api-access-q4s6t\") pod \"network-check-source-685456655f-n82mx\" (UID: \"5f7af3e7-4253-4adc-9187-05a252402917\") " pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" Sep 06 13:58:05.428035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427949 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e557111-94a1-498c-a4f9-273705d374be-serving-cert\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.428035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.427997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4r4\" (UniqueName: \"kubernetes.io/projected/7be59bd6-458a-47f0-9e1f-4e22c97a8553-kube-api-access-mz4r4\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.428279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428049 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b13bd22-de6a-4432-92d0-8990aee7c811-tmp-dir\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.428279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428079 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k2d\" (UniqueName: \"kubernetes.io/projected/7dea50e1-1e6c-4f47-960c-6a53737fb691-kube-api-access-d9k2d\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.428279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428106 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dea50e1-1e6c-4f47-960c-6a53737fb691-serving-cert\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.428279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428135 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvkc\" (UniqueName: \"kubernetes.io/projected/0a63c936-81b4-428d-b609-83a70a3e2ac6-kube-api-access-9tvkc\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.428279 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428172 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13bd22-de6a-4432-92d0-8990aee7c811-config-volume\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.428279 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.428252 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 13:58:05.428575 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.428305 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls podName:0b13bd22-de6a-4432-92d0-8990aee7c811 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.928288638 +0000 UTC m=+35.191733352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls") pod "dns-default-pd4sj" (UID: "0b13bd22-de6a-4432-92d0-8990aee7c811") : secret "dns-default-metrics-tls" not found Sep 06 13:58:05.428575 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428302 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e557111-94a1-498c-a4f9-273705d374be-config\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.428737 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428586 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b13bd22-de6a-4432-92d0-8990aee7c811-tmp-dir\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.428737 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428609 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dea50e1-1e6c-4f47-960c-6a53737fb691-config\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.428737 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.428664 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 13:58:05.428737 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.428709 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls podName:0a63c936-81b4-428d-b609-83a70a3e2ac6 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:05.928694645 +0000 UTC m=+35.192139354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cfdcf" (UID: "0a63c936-81b4-428d-b609-83a70a3e2ac6") : secret "samples-operator-tls" not found Sep 06 13:58:05.428964 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.428874 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7be59bd6-458a-47f0-9e1f-4e22c97a8553-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.429609 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.429561 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e557111-94a1-498c-a4f9-273705d374be-trusted-ca\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.430621 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.430568 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7be59bd6-458a-47f0-9e1f-4e22c97a8553-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.431068 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.431046 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e557111-94a1-498c-a4f9-273705d374be-serving-cert\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.431334 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.431310 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dea50e1-1e6c-4f47-960c-6a53737fb691-serving-cert\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.435670 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.435624 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95cv\" (UniqueName: \"kubernetes.io/projected/0e557111-94a1-498c-a4f9-273705d374be-kube-api-access-b95cv\") pod \"console-operator-85b454874b-ss4gj\" (UID: \"0e557111-94a1-498c-a4f9-273705d374be\") " pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.435802 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.435723 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6c9l\" (UniqueName: \"kubernetes.io/projected/0b13bd22-de6a-4432-92d0-8990aee7c811-kube-api-access-g6c9l\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.436536 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.436514 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvkc\" (UniqueName: \"kubernetes.io/projected/0a63c936-81b4-428d-b609-83a70a3e2ac6-kube-api-access-9tvkc\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.436633 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.436598 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4r4\" (UniqueName: \"kubernetes.io/projected/7be59bd6-458a-47f0-9e1f-4e22c97a8553-kube-api-access-mz4r4\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7qqkq\" (UID: \"7be59bd6-458a-47f0-9e1f-4e22c97a8553\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.437231 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.437188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4s6t\" (UniqueName: \"kubernetes.io/projected/5f7af3e7-4253-4adc-9187-05a252402917-kube-api-access-q4s6t\") pod \"network-check-source-685456655f-n82mx\" (UID: \"5f7af3e7-4253-4adc-9187-05a252402917\") " pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" Sep 06 13:58:05.437803 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.437781 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k2d\" (UniqueName: \"kubernetes.io/projected/7dea50e1-1e6c-4f47-960c-6a53737fb691-kube-api-access-d9k2d\") pod \"service-ca-operator-859f94c4b-bs6bh\" (UID: \"7dea50e1-1e6c-4f47-960c-6a53737fb691\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.466021 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.465981 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" Sep 06 13:58:05.501373 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.501338 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:05.518191 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.518165 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" Sep 06 13:58:05.529554 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.529519 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jqq\" (UniqueName: \"kubernetes.io/projected/ce47be1d-02c9-4e8b-866b-2531c424bc2c-kube-api-access-f8jqq\") pod \"ingress-canary-s6hq6\" (UID: \"ce47be1d-02c9-4e8b-866b-2531c424bc2c\") " pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.529705 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.529656 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qj4z\" (UniqueName: \"kubernetes.io/projected/46f7ff31-5a17-4585-b2f9-a2d4981f00d4-kube-api-access-7qj4z\") pod \"volume-data-source-validator-695c895bd6-b6w52\" (UID: \"46f7ff31-5a17-4585-b2f9-a2d4981f00d4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" Sep 06 13:58:05.529705 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.529684 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce47be1d-02c9-4e8b-866b-2531c424bc2c-cert\") pod \"ingress-canary-s6hq6\" (UID: \"ce47be1d-02c9-4e8b-866b-2531c424bc2c\") " pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.532243 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.532219 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce47be1d-02c9-4e8b-866b-2531c424bc2c-cert\") pod \"ingress-canary-s6hq6\" (UID: \"ce47be1d-02c9-4e8b-866b-2531c424bc2c\") " pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.537253 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.537230 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jqq\" (UniqueName: \"kubernetes.io/projected/ce47be1d-02c9-4e8b-866b-2531c424bc2c-kube-api-access-f8jqq\") pod \"ingress-canary-s6hq6\" (UID: \"ce47be1d-02c9-4e8b-866b-2531c424bc2c\") " pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.537371 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.537348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qj4z\" (UniqueName: \"kubernetes.io/projected/46f7ff31-5a17-4585-b2f9-a2d4981f00d4-kube-api-access-7qj4z\") pod \"volume-data-source-validator-695c895bd6-b6w52\" (UID: \"46f7ff31-5a17-4585-b2f9-a2d4981f00d4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" Sep 06 13:58:05.571532 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.571491 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" Sep 06 13:58:05.577743 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.577703 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" Sep 06 13:58:05.586690 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.586663 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" Sep 06 13:58:05.602438 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.602401 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6hq6" Sep 06 13:58:05.832454 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.832370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.832454 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.832429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:05.832454 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.832453 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:05.832718 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832528 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:05.832718 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832530 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 13:58:05.832718 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832585 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls podName:019adef7-a1fb-4778-8880-2b206ee04a44 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:06.832565626 +0000 UTC m=+36.096010331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gscwv" (UID: "019adef7-a1fb-4778-8880-2b206ee04a44") : secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:05.832718 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832604 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:06.832594806 +0000 UTC m=+36.096039512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : configmap references non-existent config key: service-ca.crt Sep 06 13:58:05.832718 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832619 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:06.832610586 +0000 UTC m=+36.096055290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : secret "router-metrics-certs-default" not found Sep 06 13:58:05.832718 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.832652 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:05.832976 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832762 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 13:58:05.832976 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832773 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7448bfcb7f-sjwzf: secret "image-registry-tls" not found Sep 06 13:58:05.832976 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.832811 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls podName:d0958c93-9d63-42d6-b109-ec5c7ae47828 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:06.832799708 +0000 UTC m=+36.096244409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls") pod "image-registry-7448bfcb7f-sjwzf" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828") : secret "image-registry-tls" not found Sep 06 13:58:05.934172 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.934131 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:05.934623 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:05.934215 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:05.934623 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.934299 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 13:58:05.934623 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.934377 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls podName:0b13bd22-de6a-4432-92d0-8990aee7c811 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:06.934356938 +0000 UTC m=+36.197801653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls") pod "dns-default-pd4sj" (UID: "0b13bd22-de6a-4432-92d0-8990aee7c811") : secret "dns-default-metrics-tls" not found Sep 06 13:58:05.934623 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.934378 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 13:58:05.934623 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:05.934446 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls podName:0a63c936-81b4-428d-b609-83a70a3e2ac6 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:06.934428267 +0000 UTC m=+36.197872971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cfdcf" (UID: "0a63c936-81b4-428d-b609-83a70a3e2ac6") : secret "samples-operator-tls" not found Sep 06 13:58:06.472747 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.472713 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6hq6"] Sep 06 13:58:06.476915 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.476822 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh"] Sep 06 13:58:06.477855 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.477651 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce47be1d_02c9_4e8b_866b_2531c424bc2c.slice/crio-86a7e0412f23d780e8062b9c7e6487bf535569f65dfaede9621498be8e4e76d7 WatchSource:0}: Error finding container 86a7e0412f23d780e8062b9c7e6487bf535569f65dfaede9621498be8e4e76d7: Status 404 returned error can't find the container with id 86a7e0412f23d780e8062b9c7e6487bf535569f65dfaede9621498be8e4e76d7 Sep 06 13:58:06.478551 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.478480 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vr22w"] Sep 06 13:58:06.479702 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.479681 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-n82mx"] Sep 06 13:58:06.481151 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.480939 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52"] Sep 06 13:58:06.481151 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.481060 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dea50e1_1e6c_4f47_960c_6a53737fb691.slice/crio-e4b5b21cbb7f31cd895b8468c728d51aab8404e1b92df15e37d0c472c55036d9 WatchSource:0}: Error finding container e4b5b21cbb7f31cd895b8468c728d51aab8404e1b92df15e37d0c472c55036d9: Status 404 returned error can't find the container with id e4b5b21cbb7f31cd895b8468c728d51aab8404e1b92df15e37d0c472c55036d9 Sep 06 13:58:06.481994 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.481976 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq"] Sep 06 13:58:06.483059 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.483026 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ss4gj"] Sep 06 13:58:06.485136 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.485106 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be59bd6_458a_47f0_9e1f_4e22c97a8553.slice/crio-7227594f487ff24f5af5ad4717e11ba8ff4af0b1d7d692d57d74e16df818e20d WatchSource:0}: Error finding container 7227594f487ff24f5af5ad4717e11ba8ff4af0b1d7d692d57d74e16df818e20d: Status 404 returned error can't find the container with id 7227594f487ff24f5af5ad4717e11ba8ff4af0b1d7d692d57d74e16df818e20d Sep 06 13:58:06.485814 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.485792 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1209f84_910d_4d92_ae80_c5eaac7cdb1f.slice/crio-4aff854961a70c8396658ed54f44d8bb6ab0babf17a511db7dcb4bc49de2b6ac WatchSource:0}: Error finding container 4aff854961a70c8396658ed54f44d8bb6ab0babf17a511db7dcb4bc49de2b6ac: Status 404 returned error can't find the container with id 4aff854961a70c8396658ed54f44d8bb6ab0babf17a511db7dcb4bc49de2b6ac Sep 06 13:58:06.486151 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.486131 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7ff31_5a17_4585_b2f9_a2d4981f00d4.slice/crio-352d4e103a10fe069c6da78d407f2d3ff377b8bfc134063dea85cf5ffc32fba0 WatchSource:0}: Error finding container 352d4e103a10fe069c6da78d407f2d3ff377b8bfc134063dea85cf5ffc32fba0: Status 404 returned error can't find the container with id 352d4e103a10fe069c6da78d407f2d3ff377b8bfc134063dea85cf5ffc32fba0 Sep 06 13:58:06.487596 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.487568 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e557111_94a1_498c_a4f9_273705d374be.slice/crio-ac0ba909ec16f8e3d03680bc8c2151be2bb6297832398121baa613e5155ed17a WatchSource:0}: Error finding container ac0ba909ec16f8e3d03680bc8c2151be2bb6297832398121baa613e5155ed17a: Status 404 returned error can't find the container with id ac0ba909ec16f8e3d03680bc8c2151be2bb6297832398121baa613e5155ed17a Sep 06 13:58:06.488584 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:06.488242 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7af3e7_4253_4adc_9187_05a252402917.slice/crio-1a8935345501c54440ae73257571968510126c2a0e6bfe3800205f651c45f535 WatchSource:0}: Error finding container 1a8935345501c54440ae73257571968510126c2a0e6bfe3800205f651c45f535: Status 404 returned error can't find the container with id 1a8935345501c54440ae73257571968510126c2a0e6bfe3800205f651c45f535 Sep 06 13:58:06.841486 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.841453 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.841529 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841600 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841665 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls podName:019adef7-a1fb-4778-8880-2b206ee04a44 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:08.841646783 +0000 UTC m=+38.105091504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gscwv" (UID: "019adef7-a1fb-4778-8880-2b206ee04a44") : secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.841650 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841676 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841735 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 13:58:06.841728 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841743 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7448bfcb7f-sjwzf: secret "image-registry-tls" not found Sep 06 13:58:06.842061 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.841722 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:06.842061 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841793 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls podName:d0958c93-9d63-42d6-b109-ec5c7ae47828 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:08.841774191 +0000 UTC m=+38.105218912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls") pod "image-registry-7448bfcb7f-sjwzf" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828") : secret "image-registry-tls" not found Sep 06 13:58:06.842061 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841816 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:08.841808339 +0000 UTC m=+38.105253045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : secret "router-metrics-certs-default" not found Sep 06 13:58:06.842061 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.841837 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:08.841822841 +0000 UTC m=+38.105267558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : configmap references non-existent config key: service-ca.crt Sep 06 13:58:06.942489 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.942454 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:06.943121 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:06.942509 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:06.943121 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.942601 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 13:58:06.943121 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.942662 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls podName:0b13bd22-de6a-4432-92d0-8990aee7c811 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:08.942647514 +0000 UTC m=+38.206092245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls") pod "dns-default-pd4sj" (UID: "0b13bd22-de6a-4432-92d0-8990aee7c811") : secret "dns-default-metrics-tls" not found Sep 06 13:58:06.943121 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.942608 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 13:58:06.943121 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:06.942726 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls podName:0a63c936-81b4-428d-b609-83a70a3e2ac6 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:08.942715434 +0000 UTC m=+38.206160135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cfdcf" (UID: "0a63c936-81b4-428d-b609-83a70a3e2ac6") : secret "samples-operator-tls" not found Sep 06 13:58:07.409176 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.408809 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" event={"ID":"5f7af3e7-4253-4adc-9187-05a252402917","Type":"ContainerStarted","Data":"1a8935345501c54440ae73257571968510126c2a0e6bfe3800205f651c45f535"} Sep 06 13:58:07.410313 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.410285 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" event={"ID":"7be59bd6-458a-47f0-9e1f-4e22c97a8553","Type":"ContainerStarted","Data":"7227594f487ff24f5af5ad4717e11ba8ff4af0b1d7d692d57d74e16df818e20d"} Sep 06 13:58:07.412126 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.412103 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" event={"ID":"b1209f84-910d-4d92-ae80-c5eaac7cdb1f","Type":"ContainerStarted","Data":"4aff854961a70c8396658ed54f44d8bb6ab0babf17a511db7dcb4bc49de2b6ac"} Sep 06 13:58:07.414215 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.414152 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" event={"ID":"0e557111-94a1-498c-a4f9-273705d374be","Type":"ContainerStarted","Data":"ac0ba909ec16f8e3d03680bc8c2151be2bb6297832398121baa613e5155ed17a"} Sep 06 13:58:07.416917 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.416838 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" event={"ID":"46f7ff31-5a17-4585-b2f9-a2d4981f00d4","Type":"ContainerStarted","Data":"352d4e103a10fe069c6da78d407f2d3ff377b8bfc134063dea85cf5ffc32fba0"} Sep 06 13:58:07.418663 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.418600 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" event={"ID":"7dea50e1-1e6c-4f47-960c-6a53737fb691","Type":"ContainerStarted","Data":"e4b5b21cbb7f31cd895b8468c728d51aab8404e1b92df15e37d0c472c55036d9"} Sep 06 13:58:07.420596 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.420534 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6hq6" event={"ID":"ce47be1d-02c9-4e8b-866b-2531c424bc2c","Type":"ContainerStarted","Data":"86a7e0412f23d780e8062b9c7e6487bf535569f65dfaede9621498be8e4e76d7"} Sep 06 13:58:07.430095 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.429048 2550 generic.go:356] "Generic (PLEG): container finished" podID="07010b81-f41e-4dd2-aced-6ce550d937ce" containerID="885f7aafbfc9759d8b34083f3750975d395da5ac4a29e53c64b6a732395d73c3" exitCode=0 Sep 06 13:58:07.430095 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:07.429094 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerDied","Data":"885f7aafbfc9759d8b34083f3750975d395da5ac4a29e53c64b6a732395d73c3"} Sep 06 13:58:08.445492 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.445455 2550 generic.go:356] "Generic (PLEG): container finished" podID="07010b81-f41e-4dd2-aced-6ce550d937ce" containerID="95b24950492a1487cb4fbdbb393c2f5e4cb998c2ab66e319e2775bbb6921f997" exitCode=0 Sep 06 13:58:08.446429 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.446286 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerDied","Data":"95b24950492a1487cb4fbdbb393c2f5e4cb998c2ab66e319e2775bbb6921f997"} Sep 06 13:58:08.864449 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.864134 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:08.864670 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864303 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:08.864670 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.864549 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:08.864670 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864586 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls podName:019adef7-a1fb-4778-8880-2b206ee04a44 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:12.864562244 +0000 UTC m=+42.128006949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gscwv" (UID: "019adef7-a1fb-4778-8880-2b206ee04a44") : secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:08.864670 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864661 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 13:58:08.864913 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864675 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7448bfcb7f-sjwzf: secret "image-registry-tls" not found Sep 06 13:58:08.864913 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.864721 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:08.864913 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864728 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls podName:d0958c93-9d63-42d6-b109-ec5c7ae47828 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:12.864711952 +0000 UTC m=+42.128156679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls") pod "image-registry-7448bfcb7f-sjwzf" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828") : secret "image-registry-tls" not found Sep 06 13:58:08.864913 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.864774 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:08.864913 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864909 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:12.864882547 +0000 UTC m=+42.128327257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : configmap references non-existent config key: service-ca.crt Sep 06 13:58:08.865185 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.864979 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 13:58:08.865185 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.865010 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:12.864999936 +0000 UTC m=+42.128444638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : secret "router-metrics-certs-default" not found Sep 06 13:58:08.966480 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.966385 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:08.966480 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:08.966464 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:08.966699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.966614 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 13:58:08.966699 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.966678 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls podName:0a63c936-81b4-428d-b609-83a70a3e2ac6 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:12.966659306 +0000 UTC m=+42.230104012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cfdcf" (UID: "0a63c936-81b4-428d-b609-83a70a3e2ac6") : secret "samples-operator-tls" not found Sep 06 13:58:08.967185 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.967103 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 13:58:08.967185 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:08.967155 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls podName:0b13bd22-de6a-4432-92d0-8990aee7c811 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:12.967138772 +0000 UTC m=+42.230583474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls") pod "dns-default-pd4sj" (UID: "0b13bd22-de6a-4432-92d0-8990aee7c811") : secret "dns-default-metrics-tls" not found Sep 06 13:58:09.454034 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:09.453979 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" event={"ID":"07010b81-f41e-4dd2-aced-6ce550d937ce","Type":"ContainerStarted","Data":"0bcf2562578a164a4fe9f0d398b8fff840f87b640dfdc7e2d86ee1efaf5000ce"} Sep 06 13:58:09.455813 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:09.455782 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" event={"ID":"46f7ff31-5a17-4585-b2f9-a2d4981f00d4","Type":"ContainerStarted","Data":"ea440eddcb12610eb0e98dc61ca7dd39359984efe232ee0f93d684aa9f5474d9"} Sep 06 13:58:09.474732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:09.474050 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lzz6f" podStartSLOduration=7.711356224 podStartE2EDuration="37.474031147s" totalImagesPullingTime="29.762674923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:09.472549783 +0000 UTC m=+38.735994511" watchObservedRunningTime="2026-09-06 13:58:09.474031147 +0000 UTC m=+38.737475872" Sep 06 13:58:09.486003 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:09.485822 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-b6w52" podStartSLOduration=11.456172361 podStartE2EDuration="13.485803745s" totalImagesPullingTime="2.029631384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:09.485543068 +0000 UTC m=+38.748987793" watchObservedRunningTime="2026-09-06 13:58:09.485803745 +0000 UTC m=+38.749248492" Sep 06 13:58:12.907150 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:12.907112 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:12.907150 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:12.907165 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:12.907193 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:12.907246 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907296 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907315 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:20.90729466 +0000 UTC m=+50.170739366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : configmap references non-existent config key: service-ca.crt Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907361 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907367 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:20.907346965 +0000 UTC m=+50.170791670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : secret "router-metrics-certs-default" not found Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907448 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls podName:019adef7-a1fb-4778-8880-2b206ee04a44 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:20.90743351 +0000 UTC m=+50.170878214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gscwv" (UID: "019adef7-a1fb-4778-8880-2b206ee04a44") : secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907367 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907476 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7448bfcb7f-sjwzf: secret "image-registry-tls" not found Sep 06 13:58:12.907714 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:12.907533 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls podName:d0958c93-9d63-42d6-b109-ec5c7ae47828 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:20.907514524 +0000 UTC m=+50.170959241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls") pod "image-registry-7448bfcb7f-sjwzf" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828") : secret "image-registry-tls" not found Sep 06 13:58:13.007935 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.007881 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:13.008120 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.007964 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:13.008120 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.008017 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 13:58:13.008120 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.008074 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 13:58:13.008120 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.008106 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls podName:0b13bd22-de6a-4432-92d0-8990aee7c811 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.008075598 +0000 UTC m=+50.271520300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls") pod "dns-default-pd4sj" (UID: "0b13bd22-de6a-4432-92d0-8990aee7c811") : secret "dns-default-metrics-tls" not found Sep 06 13:58:13.008307 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.008140 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls podName:0a63c936-81b4-428d-b609-83a70a3e2ac6 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.008124635 +0000 UTC m=+50.271569360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cfdcf" (UID: "0a63c936-81b4-428d-b609-83a70a3e2ac6") : secret "samples-operator-tls" not found Sep 06 13:58:13.105908 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.105861 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-q6dld"] Sep 06 13:58:13.138366 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.138329 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-q6dld"] Sep 06 13:58:13.138546 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.138453 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.140797 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.140772 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 06 13:58:13.141189 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.141171 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6tkjk\"" Sep 06 13:58:13.141404 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.141381 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 06 13:58:13.310640 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.310599 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.310640 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.310645 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93551ec1-12b7-43cf-85bd-d63119d7087a-nginx-conf\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.411360 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.411319 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.411360 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.411370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93551ec1-12b7-43cf-85bd-d63119d7087a-nginx-conf\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.411586 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.411492 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 13:58:13.411586 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.411564 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert podName:93551ec1-12b7-43cf-85bd-d63119d7087a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:13.911544663 +0000 UTC m=+43.174989378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q6dld" (UID: "93551ec1-12b7-43cf-85bd-d63119d7087a") : secret "networking-console-plugin-cert" not found Sep 06 13:58:13.412087 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.412063 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93551ec1-12b7-43cf-85bd-d63119d7087a-nginx-conf\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.916154 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:13.916116 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:13.916487 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.916274 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 13:58:13.916487 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:13.916346 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert podName:93551ec1-12b7-43cf-85bd-d63119d7087a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:14.916329864 +0000 UTC m=+44.179774566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q6dld" (UID: "93551ec1-12b7-43cf-85bd-d63119d7087a") : secret "networking-console-plugin-cert" not found Sep 06 13:58:14.485225 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.485190 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/0.log" Sep 06 13:58:14.485421 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.485245 2550 generic.go:356] "Generic (PLEG): container finished" podID="0e557111-94a1-498c-a4f9-273705d374be" containerID="f3136dc813778b7d2fbfbd1d31452f52c04b2988ea509dc46eaf6eae6d576cf4" exitCode=255 Sep 06 13:58:14.485421 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.485345 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" event={"ID":"0e557111-94a1-498c-a4f9-273705d374be","Type":"ContainerDied","Data":"f3136dc813778b7d2fbfbd1d31452f52c04b2988ea509dc46eaf6eae6d576cf4"} Sep 06 13:58:14.485592 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.485571 2550 scope.go:117] "RemoveContainer" containerID="f3136dc813778b7d2fbfbd1d31452f52c04b2988ea509dc46eaf6eae6d576cf4" Sep 06 13:58:14.487705 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.487678 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" event={"ID":"7dea50e1-1e6c-4f47-960c-6a53737fb691","Type":"ContainerStarted","Data":"22421318840cee6808b4b904bcc22b8462bbded09d6f904da6f3a0e3f1d2198c"} Sep 06 13:58:14.490155 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.490128 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6hq6" event={"ID":"ce47be1d-02c9-4e8b-866b-2531c424bc2c","Type":"ContainerStarted","Data":"fa1651ae5f832c07f03e2374714f9f08a580fa36320395bcf8cf326caed701d3"} Sep 06 13:58:14.492318 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.492291 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" event={"ID":"5f7af3e7-4253-4adc-9187-05a252402917","Type":"ContainerStarted","Data":"6f9bd687395cb6c7e8f88c5748057be57f7ca9c19a8ca53a8da0110e669855cb"} Sep 06 13:58:14.495556 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.495526 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" event={"ID":"7be59bd6-458a-47f0-9e1f-4e22c97a8553","Type":"ContainerStarted","Data":"b1dc04079a69dc494890c55dd4ee1cd075982a316b0274da6934dc1f0e0e9b8f"} Sep 06 13:58:14.497381 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.497354 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" event={"ID":"b1209f84-910d-4d92-ae80-c5eaac7cdb1f","Type":"ContainerStarted","Data":"d890b00cbc9d5a0dafd19543314f5e9601d32173587b921ec8ef37b56a4921bc"} Sep 06 13:58:14.517473 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.517414 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-n82mx" podStartSLOduration=10.882206147 podStartE2EDuration="18.517378881s" totalImagesPullingTime="7.635172734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:14.516249687 +0000 UTC m=+43.779694409" watchObservedRunningTime="2026-09-06 13:58:14.517378881 +0000 UTC m=+43.780823600" Sep 06 13:58:14.533922 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.532387 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" podStartSLOduration=10.900973564 podStartE2EDuration="18.532366472s" totalImagesPullingTime="7.631392908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:14.531211946 +0000 UTC m=+43.794656684" watchObservedRunningTime="2026-09-06 13:58:14.532366472 +0000 UTC m=+43.795811196" Sep 06 13:58:14.547214 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.547150 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" podStartSLOduration=10.912048502 podStartE2EDuration="18.547125148s" totalImagesPullingTime="7.635076646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:14.545623781 +0000 UTC m=+43.809068509" watchObservedRunningTime="2026-09-06 13:58:14.547125148 +0000 UTC m=+43.810569873" Sep 06 13:58:14.583630 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.581503 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s6hq6" podStartSLOduration=1.944072698 podStartE2EDuration="9.581479003s" totalImagesPullingTime="7.637406305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:14.563336454 +0000 UTC m=+43.826781176" watchObservedRunningTime="2026-09-06 13:58:14.581479003 +0000 UTC m=+43.844923729" Sep 06 13:58:14.927339 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:14.927297 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:14.927825 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:14.927489 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 13:58:14.927825 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:14.927571 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert podName:93551ec1-12b7-43cf-85bd-d63119d7087a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:16.927549637 +0000 UTC m=+46.190994346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q6dld" (UID: "93551ec1-12b7-43cf-85bd-d63119d7087a") : secret "networking-console-plugin-cert" not found Sep 06 13:58:15.462161 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.462106 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" podStartSLOduration=11.832860659 podStartE2EDuration="19.462084768s" totalImagesPullingTime="7.629224109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:14.584086287 +0000 UTC m=+43.847531013" watchObservedRunningTime="2026-09-06 13:58:15.462084768 +0000 UTC m=+44.725529486" Sep 06 13:58:15.462512 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.462497 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kvrcs"] Sep 06 13:58:15.479551 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.479485 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kvrcs"] Sep 06 13:58:15.479735 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.479640 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.481873 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.481837 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 06 13:58:15.482508 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.482488 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 06 13:58:15.482646 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.482492 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-58qh8\"" Sep 06 13:58:15.501469 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.501443 2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:15.501649 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.501492 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:15.503589 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.503566 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/1.log" Sep 06 13:58:15.504075 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.504056 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/0.log" Sep 06 13:58:15.504179 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.504088 2550 generic.go:356] "Generic (PLEG): container finished" podID="0e557111-94a1-498c-a4f9-273705d374be" containerID="96875acc8e55ccfbc716cda31f4b46393d7b11c064702bd0b7942b11ee60825f" exitCode=255 Sep 06 13:58:15.504922 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.504878 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" event={"ID":"0e557111-94a1-498c-a4f9-273705d374be","Type":"ContainerDied","Data":"96875acc8e55ccfbc716cda31f4b46393d7b11c064702bd0b7942b11ee60825f"} Sep 06 13:58:15.505077 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.505063 2550 scope.go:117] "RemoveContainer" containerID="f3136dc813778b7d2fbfbd1d31452f52c04b2988ea509dc46eaf6eae6d576cf4" Sep 06 13:58:15.505262 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.505246 2550 scope.go:117] "RemoveContainer" containerID="96875acc8e55ccfbc716cda31f4b46393d7b11c064702bd0b7942b11ee60825f" Sep 06 13:58:15.505550 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:15.505524 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ss4gj_openshift-console-operator(0e557111-94a1-498c-a4f9-273705d374be)\"" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" podUID="0e557111-94a1-498c-a4f9-273705d374be" Sep 06 13:58:15.633629 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.633587 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.634223 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.634190 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ps4\" (UniqueName: \"kubernetes.io/projected/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-api-access-x4ps4\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.634676 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.634533 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afd0bcff-9417-4f98-af61-50c78ce4f089-crio-socket\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.634676 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.634610 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.634841 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.634771 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afd0bcff-9417-4f98-af61-50c78ce4f089-data-volume\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735474 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afd0bcff-9417-4f98-af61-50c78ce4f089-data-volume\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735474 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735431 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735693 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735481 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ps4\" (UniqueName: \"kubernetes.io/projected/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-api-access-x4ps4\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735693 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:15.735570 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 13:58:15.735693 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:15.735639 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:16.235623572 +0000 UTC m=+45.499068294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : configmap "kube-rbac-proxy" not found Sep 06 13:58:15.735837 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afd0bcff-9417-4f98-af61-50c78ce4f089-crio-socket\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735837 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735742 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735837 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735765 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afd0bcff-9417-4f98-af61-50c78ce4f089-data-volume\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735964 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.735873 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afd0bcff-9417-4f98-af61-50c78ce4f089-crio-socket\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.735964 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:15.735933 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 13:58:15.736022 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:15.735984 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:16.235971054 +0000 UTC m=+45.499415757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : secret "insights-runtime-extractor-tls" not found Sep 06 13:58:15.745939 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.745910 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ps4\" (UniqueName: \"kubernetes.io/projected/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-api-access-x4ps4\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:15.891045 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.891002 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8"] Sep 06 13:58:15.923914 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.923864 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8"] Sep 06 13:58:15.924081 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.924009 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" Sep 06 13:58:15.926219 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.926195 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 06 13:58:15.926385 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.926264 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 06 13:58:15.926436 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:15.926414 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kbltb\"" Sep 06 13:58:16.039002 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.038926 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrmj\" (UniqueName: \"kubernetes.io/projected/379fa1ea-74c0-4b8c-97a9-279f8d70f952-kube-api-access-sfrmj\") pod \"migrator-fdbb8fdff-5wbh8\" (UID: \"379fa1ea-74c0-4b8c-97a9-279f8d70f952\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" Sep 06 13:58:16.139969 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.139853 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrmj\" (UniqueName: \"kubernetes.io/projected/379fa1ea-74c0-4b8c-97a9-279f8d70f952-kube-api-access-sfrmj\") pod \"migrator-fdbb8fdff-5wbh8\" (UID: \"379fa1ea-74c0-4b8c-97a9-279f8d70f952\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" Sep 06 13:58:16.147433 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.147388 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrmj\" (UniqueName: \"kubernetes.io/projected/379fa1ea-74c0-4b8c-97a9-279f8d70f952-kube-api-access-sfrmj\") pod \"migrator-fdbb8fdff-5wbh8\" (UID: \"379fa1ea-74c0-4b8c-97a9-279f8d70f952\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" Sep 06 13:58:16.233711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.233674 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" Sep 06 13:58:16.241213 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.241178 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:16.241356 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.241255 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:16.241356 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.241340 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 13:58:16.241425 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.241391 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:17.241376851 +0000 UTC m=+46.504821553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : configmap "kube-rbac-proxy" not found Sep 06 13:58:16.241425 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.241341 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 13:58:16.241505 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.241460 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:17.241446938 +0000 UTC m=+46.504891660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : secret "insights-runtime-extractor-tls" not found Sep 06 13:58:16.508809 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.508777 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/1.log" Sep 06 13:58:16.509138 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.509123 2550 scope.go:117] "RemoveContainer" containerID="96875acc8e55ccfbc716cda31f4b46393d7b11c064702bd0b7942b11ee60825f" Sep 06 13:58:16.509289 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.509271 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ss4gj_openshift-console-operator(0e557111-94a1-498c-a4f9-273705d374be)\"" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" podUID="0e557111-94a1-498c-a4f9-273705d374be" Sep 06 13:58:16.532839 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.532806 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8"] Sep 06 13:58:16.535976 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:16.535944 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379fa1ea_74c0_4b8c_97a9_279f8d70f952.slice/crio-95ca72a439c9eed79d0b8a957bc362fa47c6c9af1a7d64b45e04e6e341f384fe WatchSource:0}: Error finding container 95ca72a439c9eed79d0b8a957bc362fa47c6c9af1a7d64b45e04e6e341f384fe: Status 404 returned error can't find the container with id 95ca72a439c9eed79d0b8a957bc362fa47c6c9af1a7d64b45e04e6e341f384fe Sep 06 13:58:16.629259 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.629179 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hv2l_1a1e2bb6-a11c-4520-af37-b832772ecd1b/dns-node-resolver/0.log" Sep 06 13:58:16.947749 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:16.947717 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:16.947935 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.947882 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 13:58:16.947981 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:16.947969 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert podName:93551ec1-12b7-43cf-85bd-d63119d7087a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:20.947952343 +0000 UTC m=+50.211397046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q6dld" (UID: "93551ec1-12b7-43cf-85bd-d63119d7087a") : secret "networking-console-plugin-cert" not found Sep 06 13:58:17.250779 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:17.250741 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:17.251245 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:17.250852 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:17.251245 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:17.250927 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 13:58:17.251245 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:17.250951 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 13:58:17.251245 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:17.251004 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:19.250984425 +0000 UTC m=+48.514429145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : secret "insights-runtime-extractor-tls" not found Sep 06 13:58:17.251245 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:17.251023 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:19.251016928 +0000 UTC m=+48.514461630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : configmap "kube-rbac-proxy" not found Sep 06 13:58:17.514610 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:17.514514 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" event={"ID":"379fa1ea-74c0-4b8c-97a9-279f8d70f952","Type":"ContainerStarted","Data":"95ca72a439c9eed79d0b8a957bc362fa47c6c9af1a7d64b45e04e6e341f384fe"} Sep 06 13:58:17.833068 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:17.832993 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nq79q_94411733-7637-47ae-a05c-30a723c94748/node-ca/0.log" Sep 06 13:58:18.235094 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:18.235072 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6hq6_ce47be1d-02c9-4e8b-866b-2531c424bc2c/serve-healthcheck-canary/0.log" Sep 06 13:58:18.520306 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:18.520271 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" event={"ID":"379fa1ea-74c0-4b8c-97a9-279f8d70f952","Type":"ContainerStarted","Data":"5961bac203d77336cbfa65020273beda38022af295b3b2520aa9cfab25b283e0"} Sep 06 13:58:18.520690 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:18.520330 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" event={"ID":"379fa1ea-74c0-4b8c-97a9-279f8d70f952","Type":"ContainerStarted","Data":"0c402332512534d0c8e8a43e183e79a704e41605c4b30552fa9900186a191dc8"} Sep 06 13:58:18.535534 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:18.535480 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5wbh8" podStartSLOduration=1.941039685 podStartE2EDuration="3.535461572s" totalImagesPullingTime="1.594421887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:18.534853799 +0000 UTC m=+47.798298523" watchObservedRunningTime="2026-09-06 13:58:18.535461572 +0000 UTC m=+47.798906319" Sep 06 13:58:19.268977 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:19.268925 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:19.269177 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:19.269039 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:19.269177 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:19.269101 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 13:58:19.269177 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:19.269173 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:23.269156457 +0000 UTC m=+52.532601174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : secret "insights-runtime-extractor-tls" not found Sep 06 13:58:19.269342 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:19.269194 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 13:58:19.269342 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:19.269266 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:23.269247234 +0000 UTC m=+52.532691950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : configmap "kube-rbac-proxy" not found Sep 06 13:58:20.984171 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:20.984128 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:20.984171 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:20.984175 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984270 2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984274 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:20.984305 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984316 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.984304223 +0000 UTC m=+66.247748929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : secret "router-metrics-certs-default" not found Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984352 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert podName:93551ec1-12b7-43cf-85bd-d63119d7087a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:28.98433785 +0000 UTC m=+58.247782553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q6dld" (UID: "93551ec1-12b7-43cf-85bd-d63119d7087a") : secret "networking-console-plugin-cert" not found Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984375 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle podName:7c679d17-50c8-4f87-b37e-fdd45dbee2bf nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.984361322 +0000 UTC m=+66.247806045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle") pod "router-default-c6c6bb47b-xc5lw" (UID: "7c679d17-50c8-4f87-b37e-fdd45dbee2bf") : configmap references non-existent config key: service-ca.crt Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:20.984408 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:20.984473 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984541 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984590 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls podName:019adef7-a1fb-4778-8880-2b206ee04a44 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.984577766 +0000 UTC m=+66.248022468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-gscwv" (UID: "019adef7-a1fb-4778-8880-2b206ee04a44") : secret "cluster-monitoring-operator-tls" not found Sep 06 13:58:20.984604 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984607 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 13:58:20.984972 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984616 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7448bfcb7f-sjwzf: secret "image-registry-tls" not found Sep 06 13:58:20.984972 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:20.984644 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls podName:d0958c93-9d63-42d6-b109-ec5c7ae47828 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:36.984631907 +0000 UTC m=+66.248076648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls") pod "image-registry-7448bfcb7f-sjwzf" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828") : secret "image-registry-tls" not found Sep 06 13:58:21.084970 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:21.084926 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:21.085129 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:21.085007 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:21.085129 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:21.085046 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 13:58:21.085129 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:21.085104 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls podName:0b13bd22-de6a-4432-92d0-8990aee7c811 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:37.08509086 +0000 UTC m=+66.348535575 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls") pod "dns-default-pd4sj" (UID: "0b13bd22-de6a-4432-92d0-8990aee7c811") : secret "dns-default-metrics-tls" not found Sep 06 13:58:21.085280 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:21.085137 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 06 13:58:21.085280 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:21.085192 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls podName:0a63c936-81b4-428d-b609-83a70a3e2ac6 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:37.085178385 +0000 UTC m=+66.348623087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cfdcf" (UID: "0a63c936-81b4-428d-b609-83a70a3e2ac6") : secret "samples-operator-tls" not found Sep 06 13:58:23.303829 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:23.303783 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:23.304242 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:23.303862 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:23.304242 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:23.303949 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 13:58:23.304242 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:23.304013 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls podName:afd0bcff-9417-4f98-af61-50c78ce4f089 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:31.303996899 +0000 UTC m=+60.567441601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kvrcs" (UID: "afd0bcff-9417-4f98-af61-50c78ce4f089") : secret "insights-runtime-extractor-tls" not found Sep 06 13:58:23.304361 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:23.304341 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afd0bcff-9417-4f98-af61-50c78ce4f089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:25.501865 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:25.501827 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:25.502406 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:25.501912 2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:25.502406 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:25.502267 2550 scope.go:117] "RemoveContainer" containerID="96875acc8e55ccfbc716cda31f4b46393d7b11c064702bd0b7942b11ee60825f" Sep 06 13:58:26.547332 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:26.547304 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 13:58:26.547792 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:26.547737 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/1.log" Sep 06 13:58:26.547792 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:26.547777 2550 generic.go:356] "Generic (PLEG): container finished" podID="0e557111-94a1-498c-a4f9-273705d374be" containerID="8affe4daf139d3f8df3e9c5ccd975e8251eba74ac28815a9c0d83b92265735c0" exitCode=255 Sep 06 13:58:26.547938 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:26.547823 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" event={"ID":"0e557111-94a1-498c-a4f9-273705d374be","Type":"ContainerDied","Data":"8affe4daf139d3f8df3e9c5ccd975e8251eba74ac28815a9c0d83b92265735c0"} Sep 06 13:58:26.547938 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:26.547885 2550 scope.go:117] "RemoveContainer" containerID="96875acc8e55ccfbc716cda31f4b46393d7b11c064702bd0b7942b11ee60825f" Sep 06 13:58:26.548239 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:26.548213 2550 scope.go:117] "RemoveContainer" containerID="8affe4daf139d3f8df3e9c5ccd975e8251eba74ac28815a9c0d83b92265735c0" Sep 06 13:58:26.548408 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:26.548392 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-ss4gj_openshift-console-operator(0e557111-94a1-498c-a4f9-273705d374be)\"" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" podUID="0e557111-94a1-498c-a4f9-273705d374be" Sep 06 13:58:27.552506 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:27.552477 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 13:58:29.058142 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:29.058096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:29.058550 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:29.058256 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 06 13:58:29.058550 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:29.058326 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert podName:93551ec1-12b7-43cf-85bd-d63119d7087a nodeName:}" failed. No retries permitted until 2026-09-06 13:58:45.058309769 +0000 UTC m=+74.321754484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q6dld" (UID: "93551ec1-12b7-43cf-85bd-d63119d7087a") : secret "networking-console-plugin-cert" not found Sep 06 13:58:31.380329 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:31.380276 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:31.382624 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:31.382604 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afd0bcff-9417-4f98-af61-50c78ce4f089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kvrcs\" (UID: \"afd0bcff-9417-4f98-af61-50c78ce4f089\") " pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:31.392745 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:31.392718 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-58qh8\"" Sep 06 13:58:31.400629 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:31.400601 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wz8jk" Sep 06 13:58:31.400759 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:31.400636 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kvrcs" Sep 06 13:58:31.672284 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:31.671293 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kvrcs"] Sep 06 13:58:31.676243 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:31.676213 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd0bcff_9417_4f98_af61_50c78ce4f089.slice/crio-e09be16582d6ffa0eed589875b2bf55193ad848f6089897846a6011c587fec08 WatchSource:0}: Error finding container e09be16582d6ffa0eed589875b2bf55193ad848f6089897846a6011c587fec08: Status 404 returned error can't find the container with id e09be16582d6ffa0eed589875b2bf55193ad848f6089897846a6011c587fec08 Sep 06 13:58:32.570874 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:32.570774 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kvrcs" event={"ID":"afd0bcff-9417-4f98-af61-50c78ce4f089","Type":"ContainerStarted","Data":"d0f136bf0ea7562b2a28dd483a624e37b0f95aebe2f813087a598741048502a7"} Sep 06 13:58:32.570874 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:32.570838 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kvrcs" event={"ID":"afd0bcff-9417-4f98-af61-50c78ce4f089","Type":"ContainerStarted","Data":"8096833e6efe505d33dc000ae9a285caab5bf9beaff88a0678e70801191403c6"} Sep 06 13:58:32.570874 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:32.570864 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kvrcs" event={"ID":"afd0bcff-9417-4f98-af61-50c78ce4f089","Type":"ContainerStarted","Data":"e09be16582d6ffa0eed589875b2bf55193ad848f6089897846a6011c587fec08"} Sep 06 13:58:34.580074 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:34.580030 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kvrcs" event={"ID":"afd0bcff-9417-4f98-af61-50c78ce4f089","Type":"ContainerStarted","Data":"e57301f961b3f3a17a0fe3cf1b1e0306ac184ddd20cd323b40ee8fe30ec61bdc"} Sep 06 13:58:34.594621 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:34.594557 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kvrcs" podStartSLOduration=17.611422594 podStartE2EDuration="19.594536381s" totalImagesPullingTime="1.983113787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:58:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:34.594397739 +0000 UTC m=+63.857842463" watchObservedRunningTime="2026-09-06 13:58:34.594536381 +0000 UTC m=+63.857981106" Sep 06 13:58:35.501877 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:35.501840 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:35.502100 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:35.501915 2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:35.502316 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:35.502299 2550 scope.go:117] "RemoveContainer" containerID="8affe4daf139d3f8df3e9c5ccd975e8251eba74ac28815a9c0d83b92265735c0" Sep 06 13:58:35.502499 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:35.502479 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-ss4gj_openshift-console-operator(0e557111-94a1-498c-a4f9-273705d374be)\"" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" podUID="0e557111-94a1-498c-a4f9-273705d374be" Sep 06 13:58:36.928523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.928479 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:36.928523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.928526 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:36.929043 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.928573 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:36.930945 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.930924 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 13:58:36.931035 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.930925 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 13:58:36.931118 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.931095 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4f9\" (UniqueName: \"kubernetes.io/projected/00f7bb8f-55ef-4d41-ad99-af3d2ba7f185-kube-api-access-vw4f9\") pod \"network-check-target-bxwfx\" (UID: \"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185\") " pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:36.941326 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.941288 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7025a487-10f2-475a-8da2-30f50c9989c0-metrics-certs\") pod \"network-metrics-daemon-tw57c\" (UID: \"7025a487-10f2-475a-8da2-30f50c9989c0\") " pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:36.941470 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:36.941374 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c61bfe89-af85-4a64-a403-860883b0890b-original-pull-secret\") pod \"global-pull-secret-syncer-b7m9q\" (UID: \"c61bfe89-af85-4a64-a403-860883b0890b\") " pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:37.029439 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.029396 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:37.029439 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.029441 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:37.029673 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.029461 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:37.029673 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.029544 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:37.030193 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.030171 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-service-ca-bundle\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:37.032013 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.031986 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c679d17-50c8-4f87-b37e-fdd45dbee2bf-metrics-certs\") pod \"router-default-c6c6bb47b-xc5lw\" (UID: \"7c679d17-50c8-4f87-b37e-fdd45dbee2bf\") " pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:37.032102 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.031995 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"image-registry-7448bfcb7f-sjwzf\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:37.032102 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.031995 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/019adef7-a1fb-4778-8880-2b206ee04a44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-gscwv\" (UID: \"019adef7-a1fb-4778-8880-2b206ee04a44\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:37.054129 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.054096 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htlhn\"" Sep 06 13:58:37.059996 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.059975 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cxw7\"" Sep 06 13:58:37.062605 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.062586 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:37.065978 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.065950 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7m9q" Sep 06 13:58:37.069058 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.069033 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tw57c" Sep 06 13:58:37.130916 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.130853 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:37.131063 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.130928 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:37.133402 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.133354 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b13bd22-de6a-4432-92d0-8990aee7c811-metrics-tls\") pod \"dns-default-pd4sj\" (UID: \"0b13bd22-de6a-4432-92d0-8990aee7c811\") " pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:37.133639 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.133601 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a63c936-81b4-428d-b609-83a70a3e2ac6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cfdcf\" (UID: \"0a63c936-81b4-428d-b609-83a70a3e2ac6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:37.221670 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.221637 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7kglp\"" Sep 06 13:58:37.229499 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.229465 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:37.233761 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.233732 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hzz5m\"" Sep 06 13:58:37.242491 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.242457 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" Sep 06 13:58:37.251836 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.251806 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2qrl5\"" Sep 06 13:58:37.260706 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.260667 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:37.285352 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.285321 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b29w2\"" Sep 06 13:58:37.293681 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.293654 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" Sep 06 13:58:37.332502 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.332437 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mvgg4\"" Sep 06 13:58:37.341808 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.340841 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:37.369323 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:37.369285 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f7bb8f_55ef_4d41_ad99_af3d2ba7f185.slice/crio-70506e8d25e75ad6a9ed126e5374df32febb8fd1c6ae09b045a2ebe06b74d576 WatchSource:0}: Error finding container 70506e8d25e75ad6a9ed126e5374df32febb8fd1c6ae09b045a2ebe06b74d576: Status 404 returned error can't find the container with id 70506e8d25e75ad6a9ed126e5374df32febb8fd1c6ae09b045a2ebe06b74d576 Sep 06 13:58:37.371930 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.371903 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bxwfx"] Sep 06 13:58:37.391672 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.391513 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7m9q"] Sep 06 13:58:37.403224 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:37.403181 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61bfe89_af85_4a64_a403_860883b0890b.slice/crio-c1de44822ef761edf05c4159792c2ecfa14ec6097f8e9bce2d3419c1f3ddae67 WatchSource:0}: Error finding container c1de44822ef761edf05c4159792c2ecfa14ec6097f8e9bce2d3419c1f3ddae67: Status 404 returned error can't find the container with id c1de44822ef761edf05c4159792c2ecfa14ec6097f8e9bce2d3419c1f3ddae67 Sep 06 13:58:37.415273 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.414984 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tw57c"] Sep 06 13:58:37.539392 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.539357 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7448bfcb7f-sjwzf"] Sep 06 13:58:37.542441 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:37.542409 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0958c93_9d63_42d6_b109_ec5c7ae47828.slice/crio-e19fef653e2d18a4dd0e605eb2f605883891269f8b680ca925ecafaad254a6ae WatchSource:0}: Error finding container e19fef653e2d18a4dd0e605eb2f605883891269f8b680ca925ecafaad254a6ae: Status 404 returned error can't find the container with id e19fef653e2d18a4dd0e605eb2f605883891269f8b680ca925ecafaad254a6ae Sep 06 13:58:37.570941 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.570885 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv"] Sep 06 13:58:37.573579 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:37.573547 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019adef7_a1fb_4778_8880_2b206ee04a44.slice/crio-56fa19a0ef95a4815a0016cb4ff3f558abc723bf0c25b0a0366bc5bd2452bc86 WatchSource:0}: Error finding container 56fa19a0ef95a4815a0016cb4ff3f558abc723bf0c25b0a0366bc5bd2452bc86: Status 404 returned error can't find the container with id 56fa19a0ef95a4815a0016cb4ff3f558abc723bf0c25b0a0366bc5bd2452bc86 Sep 06 13:58:37.592103 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.592069 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7m9q" event={"ID":"c61bfe89-af85-4a64-a403-860883b0890b","Type":"ContainerStarted","Data":"c1de44822ef761edf05c4159792c2ecfa14ec6097f8e9bce2d3419c1f3ddae67"} Sep 06 13:58:37.593677 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.593651 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bxwfx" event={"ID":"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185","Type":"ContainerStarted","Data":"5f28ccea862ce29fd397971a0a830294c4d89a6eebf93757830bacef47a71f3f"} Sep 06 13:58:37.593782 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.593692 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bxwfx" event={"ID":"00f7bb8f-55ef-4d41-ad99-af3d2ba7f185","Type":"ContainerStarted","Data":"70506e8d25e75ad6a9ed126e5374df32febb8fd1c6ae09b045a2ebe06b74d576"} Sep 06 13:58:37.593782 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.593764 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:58:37.595055 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.595033 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" event={"ID":"d0958c93-9d63-42d6-b109-ec5c7ae47828","Type":"ContainerStarted","Data":"e19fef653e2d18a4dd0e605eb2f605883891269f8b680ca925ecafaad254a6ae"} Sep 06 13:58:37.596692 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.596665 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" event={"ID":"019adef7-a1fb-4778-8880-2b206ee04a44","Type":"ContainerStarted","Data":"56fa19a0ef95a4815a0016cb4ff3f558abc723bf0c25b0a0366bc5bd2452bc86"} Sep 06 13:58:37.598062 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.598040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tw57c" event={"ID":"7025a487-10f2-475a-8da2-30f50c9989c0","Type":"ContainerStarted","Data":"78f4b18cb3c86a925a1a27b93cdc9657153bfaa51eb299ddf6f26d65ca90c89e"} Sep 06 13:58:37.602323 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.602296 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c6c6bb47b-xc5lw"] Sep 06 13:58:37.606000 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:37.605969 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c679d17_50c8_4f87_b37e_fdd45dbee2bf.slice/crio-47e59576cf2ad8300ea9c17b95b62bbc50353ef3fd57823b257ce733bb7d4cdc WatchSource:0}: Error finding container 47e59576cf2ad8300ea9c17b95b62bbc50353ef3fd57823b257ce733bb7d4cdc: Status 404 returned error can't find the container with id 47e59576cf2ad8300ea9c17b95b62bbc50353ef3fd57823b257ce733bb7d4cdc Sep 06 13:58:37.609960 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.609915 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bxwfx" podStartSLOduration=65.609884433 podStartE2EDuration="1m5.609884433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:37.609572351 +0000 UTC m=+66.873017077" watchObservedRunningTime="2026-09-06 13:58:37.609884433 +0000 UTC m=+66.873329177" Sep 06 13:58:37.613858 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.613829 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf"] Sep 06 13:58:37.642340 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:37.642316 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pd4sj"] Sep 06 13:58:37.646516 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:37.646490 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b13bd22_de6a_4432_92d0_8990aee7c811.slice/crio-7eb8bef98419f9f5b0290aa8f08da0643cc94de1dd3873332f564d15c472c926 WatchSource:0}: Error finding container 7eb8bef98419f9f5b0290aa8f08da0643cc94de1dd3873332f564d15c472c926: Status 404 returned error can't find the container with id 7eb8bef98419f9f5b0290aa8f08da0643cc94de1dd3873332f564d15c472c926 Sep 06 13:58:38.607164 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.607124 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" event={"ID":"d0958c93-9d63-42d6-b109-ec5c7ae47828","Type":"ContainerStarted","Data":"da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972"} Sep 06 13:58:38.607965 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.607942 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:58:38.609652 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.609594 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pd4sj" event={"ID":"0b13bd22-de6a-4432-92d0-8990aee7c811","Type":"ContainerStarted","Data":"7eb8bef98419f9f5b0290aa8f08da0643cc94de1dd3873332f564d15c472c926"} Sep 06 13:58:38.612879 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.611990 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" event={"ID":"0a63c936-81b4-428d-b609-83a70a3e2ac6","Type":"ContainerStarted","Data":"f1b8daba9eb6d2ae2757349a9d177b914961275dc8a9be2e7b3ab1333c6ffa47"} Sep 06 13:58:38.616844 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.616808 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" event={"ID":"7c679d17-50c8-4f87-b37e-fdd45dbee2bf","Type":"ContainerStarted","Data":"34ba6b6732a754a9e46a3a2150816504bbb822637e64b19e3dda3672265db02a"} Sep 06 13:58:38.617025 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.616861 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" event={"ID":"7c679d17-50c8-4f87-b37e-fdd45dbee2bf","Type":"ContainerStarted","Data":"47e59576cf2ad8300ea9c17b95b62bbc50353ef3fd57823b257ce733bb7d4cdc"} Sep 06 13:58:38.631242 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.631177 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" podStartSLOduration=66.631152891 podStartE2EDuration="1m6.631152891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:38.628809659 +0000 UTC m=+67.892254383" watchObservedRunningTime="2026-09-06 13:58:38.631152891 +0000 UTC m=+67.894597617" Sep 06 13:58:38.646590 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:38.646530 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" podStartSLOduration=42.646506001 podStartE2EDuration="42.646506001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:38.645263653 +0000 UTC m=+67.908708384" watchObservedRunningTime="2026-09-06 13:58:38.646506001 +0000 UTC m=+67.909950825" Sep 06 13:58:39.261721 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.261685 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:39.264670 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.264585 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:39.622021 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.621922 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tw57c" event={"ID":"7025a487-10f2-475a-8da2-30f50c9989c0","Type":"ContainerStarted","Data":"96ead28d5f499a413e36c41766cc72c0c8dd06e8d78158350ca949982c960eb0"} Sep 06 13:58:39.622021 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.621990 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tw57c" event={"ID":"7025a487-10f2-475a-8da2-30f50c9989c0","Type":"ContainerStarted","Data":"d09805816ef29a05e0d69ec349f62e6d639b05e944933484dd1c39f8d0e2cfc0"} Sep 06 13:58:39.622665 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.622639 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:39.624034 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.624008 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c6c6bb47b-xc5lw" Sep 06 13:58:39.656604 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:39.656510 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tw57c" podStartSLOduration=66.441653832 podStartE2EDuration="1m7.656486143s" totalImagesPullingTime="1.214832311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:39.636621359 +0000 UTC m=+68.900066099" watchObservedRunningTime="2026-09-06 13:58:39.656486143 +0000 UTC m=+68.919930869" Sep 06 13:58:42.639114 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.639028 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" event={"ID":"019adef7-a1fb-4778-8880-2b206ee04a44","Type":"ContainerStarted","Data":"5e798c8a38e2d0e4917eaa92af1416dbef46f3eef6ada754d3f8570d3a29728b"} Sep 06 13:58:42.641812 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.641777 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7m9q" event={"ID":"c61bfe89-af85-4a64-a403-860883b0890b","Type":"ContainerStarted","Data":"670578648d311203ac2a66bc0482203ae49c99fd797996c9ae9d5c8b7b9e9796"} Sep 06 13:58:42.643687 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.643663 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pd4sj" event={"ID":"0b13bd22-de6a-4432-92d0-8990aee7c811","Type":"ContainerStarted","Data":"2a249e2bbc1ea8317ff3f937b8346684de0084239ea8cdd86422d4ad8959f498"} Sep 06 13:58:42.643854 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.643840 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:42.643986 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.643971 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pd4sj" event={"ID":"0b13bd22-de6a-4432-92d0-8990aee7c811","Type":"ContainerStarted","Data":"a689d903f44910e06155ec31af7faf57307c02b16c95890454a162d295b931b0"} Sep 06 13:58:42.646221 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.646199 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" event={"ID":"0a63c936-81b4-428d-b609-83a70a3e2ac6","Type":"ContainerStarted","Data":"607d69a143bda3e017e289fbb21f63dc1d7a59db83c2c7e6bc1df8a18c633142"} Sep 06 13:58:42.646322 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.646244 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" event={"ID":"0a63c936-81b4-428d-b609-83a70a3e2ac6","Type":"ContainerStarted","Data":"d7e7af77b4332c5e8ecee68be383f63767c5efc73fe1a2c3d6a199aa9c04665d"} Sep 06 13:58:42.657023 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.656975 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-gscwv" podStartSLOduration=41.975540022 podStartE2EDuration="46.656956265s" totalImagesPullingTime="4.681416243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:42.655986577 +0000 UTC m=+71.919431302" watchObservedRunningTime="2026-09-06 13:58:42.656956265 +0000 UTC m=+71.920400994" Sep 06 13:58:42.670563 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.670512 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b7m9q" podStartSLOduration=65.758252534 podStartE2EDuration="1m10.67049829s" totalImagesPullingTime="4.912245756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:42.670443523 +0000 UTC m=+71.933888247" watchObservedRunningTime="2026-09-06 13:58:42.67049829 +0000 UTC m=+71.933943010" Sep 06 13:58:42.693833 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.693763 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pd4sj" podStartSLOduration=33.082591676 podStartE2EDuration="37.693741969s" totalImagesPullingTime="4.611150293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:42.693291599 +0000 UTC m=+71.956736323" watchObservedRunningTime="2026-09-06 13:58:42.693741969 +0000 UTC m=+71.957186697" Sep 06 13:58:42.708060 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:42.708015 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cfdcf" podStartSLOduration=42.132792522 podStartE2EDuration="46.707997261s" totalImagesPullingTime="4.575204739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:42.707232181 +0000 UTC m=+71.970676908" watchObservedRunningTime="2026-09-06 13:58:42.707997261 +0000 UTC m=+71.971441985" Sep 06 13:58:45.106214 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:45.106164 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:45.108655 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:45.108632 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93551ec1-12b7-43cf-85bd-d63119d7087a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q6dld\" (UID: \"93551ec1-12b7-43cf-85bd-d63119d7087a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:45.250885 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:45.250853 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6tkjk\"" Sep 06 13:58:45.259546 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:45.259517 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" Sep 06 13:58:45.558010 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:45.557968 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-q6dld"] Sep 06 13:58:45.561036 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:45.561009 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93551ec1_12b7_43cf_85bd_d63119d7087a.slice/crio-0d4895897688267e6b6169e87b89f1cd80baf023f28eed4eac486dab24a7cf2a WatchSource:0}: Error finding container 0d4895897688267e6b6169e87b89f1cd80baf023f28eed4eac486dab24a7cf2a: Status 404 returned error can't find the container with id 0d4895897688267e6b6169e87b89f1cd80baf023f28eed4eac486dab24a7cf2a Sep 06 13:58:45.659561 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:45.659523 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" event={"ID":"93551ec1-12b7-43cf-85bd-d63119d7087a","Type":"ContainerStarted","Data":"0d4895897688267e6b6169e87b89f1cd80baf023f28eed4eac486dab24a7cf2a"} Sep 06 13:58:47.669231 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:47.669199 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" event={"ID":"93551ec1-12b7-43cf-85bd-d63119d7087a","Type":"ContainerStarted","Data":"df902ade70c3b922585ab31fef7d323e7aba6a63a99757c891dd67b4689a9b24"} Sep 06 13:58:47.685732 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:47.685677 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-q6dld" podStartSLOduration=32.953896003 podStartE2EDuration="34.685659952s" totalImagesPullingTime="1.731763949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:47.684863443 +0000 UTC m=+76.948308177" watchObservedRunningTime="2026-09-06 13:58:47.685659952 +0000 UTC m=+76.949104749" Sep 06 13:58:48.230051 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:48.230025 2550 scope.go:117] "RemoveContainer" containerID="8affe4daf139d3f8df3e9c5ccd975e8251eba74ac28815a9c0d83b92265735c0" Sep 06 13:58:48.675144 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:48.675115 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 13:58:48.675534 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:48.675222 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" event={"ID":"0e557111-94a1-498c-a4f9-273705d374be","Type":"ContainerStarted","Data":"cb2dfbeee528a9f86e0701eebcc78da69e76180d78cc688c0245d7730aa3b619"} Sep 06 13:58:48.675634 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:48.675616 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:48.691098 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:48.691042 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" podStartSLOduration=45.061712732 podStartE2EDuration="52.691022465s" totalImagesPullingTime="7.629309733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:48.690083967 +0000 UTC m=+77.953528693" watchObservedRunningTime="2026-09-06 13:58:48.691022465 +0000 UTC m=+77.954467189" Sep 06 13:58:48.988639 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:48.988562 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-ss4gj" Sep 06 13:58:50.231204 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.231169 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pzff7"] Sep 06 13:58:50.233677 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.233660 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.235833 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.235805 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 06 13:58:50.235990 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.235885 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zxjvq\"" Sep 06 13:58:50.236537 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.236518 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 06 13:58:50.236578 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.236518 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 06 13:58:50.236631 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.236544 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 06 13:58:50.348511 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348476 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-root\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348511 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348513 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-wtmp\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348735 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348536 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-textfile\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348735 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348598 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348735 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348651 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvnj\" (UniqueName: \"kubernetes.io/projected/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-kube-api-access-vhvnj\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348735 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348684 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-sys\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348735 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348703 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-metrics-client-ca\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348946 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348748 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-accelerators-collector-config\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.348946 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.348776 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-tls\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449461 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449420 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-accelerators-collector-config\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449461 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449466 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-tls\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449497 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-root\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449515 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-wtmp\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449575 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-textfile\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449628 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:50.449650 2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449660 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvnj\" (UniqueName: \"kubernetes.io/projected/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-kube-api-access-vhvnj\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.449711 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449582 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-root\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450050 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:58:50.449726 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-tls podName:3cc3ddb6-10e2-4ad6-9d66-90f36e734a26 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:50.949705584 +0000 UTC m=+80.213150288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-tls") pod "node-exporter-pzff7" (UID: "3cc3ddb6-10e2-4ad6-9d66-90f36e734a26") : secret "node-exporter-tls" not found Sep 06 13:58:50.450050 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449743 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-sys\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450050 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449772 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-metrics-client-ca\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450050 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449806 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-sys\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450050 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449865 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-wtmp\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450050 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.449885 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-textfile\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450253 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.450129 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-accelerators-collector-config\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.450284 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.450276 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-metrics-client-ca\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.452092 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.452073 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.461198 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.461164 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvnj\" (UniqueName: \"kubernetes.io/projected/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-kube-api-access-vhvnj\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.952803 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.952767 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-tls\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:50.955378 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:50.955347 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cc3ddb6-10e2-4ad6-9d66-90f36e734a26-node-exporter-tls\") pod \"node-exporter-pzff7\" (UID: \"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26\") " pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:51.143180 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:51.143145 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pzff7" Sep 06 13:58:51.153346 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:58:51.153316 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc3ddb6_10e2_4ad6_9d66_90f36e734a26.slice/crio-aafde5a4051a767ff2b31675fac2bb47e3bc63620730fa595120eed603297e5b WatchSource:0}: Error finding container aafde5a4051a767ff2b31675fac2bb47e3bc63620730fa595120eed603297e5b: Status 404 returned error can't find the container with id aafde5a4051a767ff2b31675fac2bb47e3bc63620730fa595120eed603297e5b Sep 06 13:58:51.688328 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:51.688287 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzff7" event={"ID":"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26","Type":"ContainerStarted","Data":"aafde5a4051a767ff2b31675fac2bb47e3bc63620730fa595120eed603297e5b"} Sep 06 13:58:52.652265 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:52.652232 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pd4sj" Sep 06 13:58:52.695117 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:52.695076 2550 generic.go:356] "Generic (PLEG): container finished" podID="3cc3ddb6-10e2-4ad6-9d66-90f36e734a26" containerID="906e1264c87dc2e61f80a7009f994b90e0b424a68e6c4e062c97c043b136c1ce" exitCode=0 Sep 06 13:58:52.695587 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:52.695160 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzff7" event={"ID":"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26","Type":"ContainerDied","Data":"906e1264c87dc2e61f80a7009f994b90e0b424a68e6c4e062c97c043b136c1ce"} Sep 06 13:58:53.701293 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:53.701174 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzff7" event={"ID":"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26","Type":"ContainerStarted","Data":"a1ab493e7fc5fff277cbaecf8193fb6f663ab17c757fb9b163789c1f131d07d5"} Sep 06 13:58:53.701293 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:53.701230 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzff7" event={"ID":"3cc3ddb6-10e2-4ad6-9d66-90f36e734a26","Type":"ContainerStarted","Data":"2fe67a6d7aa9ea68f68877acac992c86e4d4fee458658b3aa57e9d9c22cdf247"} Sep 06 13:58:53.718508 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:58:53.718455 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pzff7" podStartSLOduration=2.791781167 podStartE2EDuration="3.718437347s" totalImagesPullingTime="862.481533ms" totalInitContainerRuntime="64.174647ms" isStatefulPod=true podCreationTimestamp="2026-09-06 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:53.717406419 +0000 UTC m=+82.980851143" watchObservedRunningTime="2026-09-06 13:58:53.718437347 +0000 UTC m=+82.981882071" Sep 06 13:59:00.630877 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:00.630845 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:59:01.100523 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.100490 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7865949959-ltzq7"] Sep 06 13:59:01.104879 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.104858 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.107070 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.107050 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 06 13:59:01.107182 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.107050 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 06 13:59:01.108158 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.108136 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 06 13:59:01.108274 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.108192 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 06 13:59:01.108344 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.108275 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m4nfh\"" Sep 06 13:59:01.108344 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.108316 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 06 13:59:01.108445 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.108397 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 06 13:59:01.108445 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.108414 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 06 13:59:01.111638 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.111613 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7865949959-ltzq7"] Sep 06 13:59:01.230282 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.230240 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-oauth-config\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.230474 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.230304 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-config\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.230474 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.230407 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fl5\" (UniqueName: \"kubernetes.io/projected/f1398bee-184b-4d0b-9056-ca03fbd96d4d-kube-api-access-58fl5\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.230474 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.230455 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-serving-cert\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.230617 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.230493 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-service-ca\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.230617 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.230514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-oauth-serving-cert\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.331616 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.331565 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fl5\" (UniqueName: \"kubernetes.io/projected/f1398bee-184b-4d0b-9056-ca03fbd96d4d-kube-api-access-58fl5\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.331616 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.331620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-serving-cert\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.331851 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.331646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-service-ca\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.331851 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.331660 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-oauth-serving-cert\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.331851 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.331678 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-oauth-config\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.331851 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.331709 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-config\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.332465 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.332432 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-service-ca\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.332586 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.332482 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-oauth-serving-cert\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.332586 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.332510 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-config\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.334251 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.334229 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-oauth-config\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.334337 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.334235 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-serving-cert\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.338482 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.338463 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fl5\" (UniqueName: \"kubernetes.io/projected/f1398bee-184b-4d0b-9056-ca03fbd96d4d-kube-api-access-58fl5\") pod \"console-7865949959-ltzq7\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.415002 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.414915 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:01.689778 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.689744 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7865949959-ltzq7"] Sep 06 13:59:01.692880 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:59:01.692853 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1398bee_184b_4d0b_9056_ca03fbd96d4d.slice/crio-4d3ea2d7a40ba82208eb3338f6abd4661c1d1a2aefbb824a2a92d35faf7d8fd2 WatchSource:0}: Error finding container 4d3ea2d7a40ba82208eb3338f6abd4661c1d1a2aefbb824a2a92d35faf7d8fd2: Status 404 returned error can't find the container with id 4d3ea2d7a40ba82208eb3338f6abd4661c1d1a2aefbb824a2a92d35faf7d8fd2 Sep 06 13:59:01.729440 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.729406 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865949959-ltzq7" event={"ID":"f1398bee-184b-4d0b-9056-ca03fbd96d4d","Type":"ContainerStarted","Data":"4d3ea2d7a40ba82208eb3338f6abd4661c1d1a2aefbb824a2a92d35faf7d8fd2"} Sep 06 13:59:01.972303 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:01.972188 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7448bfcb7f-sjwzf"] Sep 06 13:59:04.743489 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:04.743446 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865949959-ltzq7" event={"ID":"f1398bee-184b-4d0b-9056-ca03fbd96d4d","Type":"ContainerStarted","Data":"65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23"} Sep 06 13:59:04.758515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:04.758470 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7865949959-ltzq7" podStartSLOduration=1.059430998 podStartE2EDuration="3.758454457s" totalImagesPullingTime="2.699023459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:04.758292513 +0000 UTC m=+94.021737238" watchObservedRunningTime="2026-09-06 13:59:04.758454457 +0000 UTC m=+94.021899180" Sep 06 13:59:08.618969 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:08.618935 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bxwfx" Sep 06 13:59:09.134189 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.134149 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59db55c7db-lhffr"] Sep 06 13:59:09.139413 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.139388 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.147029 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.146996 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 06 13:59:09.147742 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.147000 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59db55c7db-lhffr"] Sep 06 13:59:09.300992 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.300959 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjgw\" (UniqueName: \"kubernetes.io/projected/5528fcb6-b444-4c63-881c-d7f54ea7363d-kube-api-access-7wjgw\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.301160 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.301004 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-trusted-ca-bundle\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.301160 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.301024 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-oauth-serving-cert\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.301160 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.301079 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-service-ca\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.301160 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.301117 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-serving-cert\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.301160 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.301145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-oauth-config\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.301346 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.301221 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-config\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.401812 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.401730 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-config\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.401812 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.401779 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjgw\" (UniqueName: \"kubernetes.io/projected/5528fcb6-b444-4c63-881c-d7f54ea7363d-kube-api-access-7wjgw\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.401812 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.401807 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-trusted-ca-bundle\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402054 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.401826 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-oauth-serving-cert\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402054 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402012 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-service-ca\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402165 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402060 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-serving-cert\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402220 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402171 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-oauth-config\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402566 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402536 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-config\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402692 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402545 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-oauth-serving-cert\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402692 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402685 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-service-ca\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.402788 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.402695 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-trusted-ca-bundle\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.404508 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.404478 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-serving-cert\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.404632 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.404610 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-oauth-config\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.411807 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.411781 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjgw\" (UniqueName: \"kubernetes.io/projected/5528fcb6-b444-4c63-881c-d7f54ea7363d-kube-api-access-7wjgw\") pod \"console-59db55c7db-lhffr\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.450621 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.450590 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:09.726086 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.726056 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59db55c7db-lhffr"] Sep 06 13:59:09.729493 ip-10-0-141-9 kubenswrapper[2550]: W0906 13:59:09.729465 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5528fcb6_b444_4c63_881c_d7f54ea7363d.slice/crio-9260ac3c2deadd0fd116de8a6c6a76e80d41a517990a5240e360b8bbd02c9b43 WatchSource:0}: Error finding container 9260ac3c2deadd0fd116de8a6c6a76e80d41a517990a5240e360b8bbd02c9b43: Status 404 returned error can't find the container with id 9260ac3c2deadd0fd116de8a6c6a76e80d41a517990a5240e360b8bbd02c9b43 Sep 06 13:59:09.764589 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:09.764554 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59db55c7db-lhffr" event={"ID":"5528fcb6-b444-4c63-881c-d7f54ea7363d","Type":"ContainerStarted","Data":"9260ac3c2deadd0fd116de8a6c6a76e80d41a517990a5240e360b8bbd02c9b43"} Sep 06 13:59:10.770122 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:10.770082 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59db55c7db-lhffr" event={"ID":"5528fcb6-b444-4c63-881c-d7f54ea7363d","Type":"ContainerStarted","Data":"7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3"} Sep 06 13:59:10.787952 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:10.787887 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59db55c7db-lhffr" podStartSLOduration=1.787843133 podStartE2EDuration="1.787843133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:59:10.786300455 +0000 UTC m=+100.049745180" watchObservedRunningTime="2026-09-06 13:59:10.787843133 +0000 UTC m=+100.051287857" Sep 06 13:59:11.416042 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:11.416001 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:11.416236 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:11.416102 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:11.421043 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:11.421016 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:11.777614 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:11.777584 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:19.450865 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:19.450826 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:19.451396 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:19.450930 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:19.455703 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:19.455676 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:19.808110 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:19.808027 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 13:59:19.849398 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:19.849365 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7865949959-ltzq7"] Sep 06 13:59:26.993457 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:26.993395 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" podUID="d0958c93-9d63-42d6-b109-ec5c7ae47828" containerName="registry" containerID="cri-o://da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972" gracePeriod=30 Sep 06 13:59:27.253251 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.253171 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:59:27.340621 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340576 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-trusted-ca\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.340798 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340638 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-bound-sa-token\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.340798 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340667 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0958c93-9d63-42d6-b109-ec5c7ae47828-ca-trust-extracted\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.340798 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340688 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-certificates\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.340798 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340713 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.340798 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340740 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-image-registry-private-configuration\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.340798 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340770 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-installation-pull-secrets\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.341204 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.340822 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqjtb\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-kube-api-access-rqjtb\") pod \"d0958c93-9d63-42d6-b109-ec5c7ae47828\" (UID: \"d0958c93-9d63-42d6-b109-ec5c7ae47828\") " Sep 06 13:59:27.341204 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.341113 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 13:59:27.341204 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.341170 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 13:59:27.343395 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.343359 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-kube-api-access-rqjtb" (OuterVolumeSpecName: "kube-api-access-rqjtb") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "kube-api-access-rqjtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 13:59:27.343542 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.343423 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 13:59:27.343542 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.343497 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 13:59:27.343706 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.343684 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 13:59:27.343742 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.343708 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 13:59:27.349390 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.349351 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0958c93-9d63-42d6-b109-ec5c7ae47828-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d0958c93-9d63-42d6-b109-ec5c7ae47828" (UID: "d0958c93-9d63-42d6-b109-ec5c7ae47828"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 13:59:27.442088 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442040 2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-bound-sa-token\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442088 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442082 2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0958c93-9d63-42d6-b109-ec5c7ae47828-ca-trust-extracted\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442088 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442093 2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-certificates\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442088 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442102 2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-registry-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442088 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442111 2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-image-registry-private-configuration\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442390 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442120 2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0958c93-9d63-42d6-b109-ec5c7ae47828-installation-pull-secrets\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442390 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442130 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqjtb\" (UniqueName: \"kubernetes.io/projected/d0958c93-9d63-42d6-b109-ec5c7ae47828-kube-api-access-rqjtb\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.442390 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.442138 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0958c93-9d63-42d6-b109-ec5c7ae47828-trusted-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:27.835693 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.835659 2550 generic.go:356] "Generic (PLEG): container finished" podID="d0958c93-9d63-42d6-b109-ec5c7ae47828" containerID="da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972" exitCode=0 Sep 06 13:59:27.835873 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.835702 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" Sep 06 13:59:27.835873 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.835714 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" event={"ID":"d0958c93-9d63-42d6-b109-ec5c7ae47828","Type":"ContainerDied","Data":"da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972"} Sep 06 13:59:27.835873 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.835757 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7448bfcb7f-sjwzf" event={"ID":"d0958c93-9d63-42d6-b109-ec5c7ae47828","Type":"ContainerDied","Data":"e19fef653e2d18a4dd0e605eb2f605883891269f8b680ca925ecafaad254a6ae"} Sep 06 13:59:27.835873 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.835778 2550 scope.go:117] "RemoveContainer" containerID="da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972" Sep 06 13:59:27.844515 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.844490 2550 scope.go:117] "RemoveContainer" containerID="da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972" Sep 06 13:59:27.844949 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:59:27.844916 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972\": container with ID starting with da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972 not found: ID does not exist" containerID="da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972" Sep 06 13:59:27.845022 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.844962 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972"} err="failed to get container status \"da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972\": rpc error: code = NotFound desc = could not find container \"da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972\": container with ID starting with da2e6729bfbb017a1b7002d70c0d6ad8e4d84dd1ef8bd2e3eb31a07a1a77d972 not found: ID does not exist" Sep 06 13:59:27.855590 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.855553 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7448bfcb7f-sjwzf"] Sep 06 13:59:27.858852 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:27.858821 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7448bfcb7f-sjwzf"] Sep 06 13:59:29.234736 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:29.234703 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0958c93-9d63-42d6-b109-ec5c7ae47828" path="/var/lib/kubelet/pods/d0958c93-9d63-42d6-b109-ec5c7ae47828/volumes" Sep 06 13:59:35.868628 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:35.868587 2550 generic.go:356] "Generic (PLEG): container finished" podID="7dea50e1-1e6c-4f47-960c-6a53737fb691" containerID="22421318840cee6808b4b904bcc22b8462bbded09d6f904da6f3a0e3f1d2198c" exitCode=0 Sep 06 13:59:35.869070 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:35.868662 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" event={"ID":"7dea50e1-1e6c-4f47-960c-6a53737fb691","Type":"ContainerDied","Data":"22421318840cee6808b4b904bcc22b8462bbded09d6f904da6f3a0e3f1d2198c"} Sep 06 13:59:35.869070 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:35.869053 2550 scope.go:117] "RemoveContainer" containerID="22421318840cee6808b4b904bcc22b8462bbded09d6f904da6f3a0e3f1d2198c" Sep 06 13:59:35.870307 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:35.870282 2550 generic.go:356] "Generic (PLEG): container finished" podID="b1209f84-910d-4d92-ae80-c5eaac7cdb1f" containerID="d890b00cbc9d5a0dafd19543314f5e9601d32173587b921ec8ef37b56a4921bc" exitCode=0 Sep 06 13:59:35.870403 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:35.870324 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" event={"ID":"b1209f84-910d-4d92-ae80-c5eaac7cdb1f","Type":"ContainerDied","Data":"d890b00cbc9d5a0dafd19543314f5e9601d32173587b921ec8ef37b56a4921bc"} Sep 06 13:59:35.870650 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:35.870628 2550 scope.go:117] "RemoveContainer" containerID="d890b00cbc9d5a0dafd19543314f5e9601d32173587b921ec8ef37b56a4921bc" Sep 06 13:59:36.876473 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:36.876428 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-bs6bh" event={"ID":"7dea50e1-1e6c-4f47-960c-6a53737fb691","Type":"ContainerStarted","Data":"13b5952ae17b113dc94a06c8d53316df96045c14bf3d9e0925585fad8091890a"} Sep 06 13:59:36.878261 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:36.878233 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vr22w" event={"ID":"b1209f84-910d-4d92-ae80-c5eaac7cdb1f","Type":"ContainerStarted","Data":"016b7e7d84d8eeb8c90dfbcfa2d6564d063eecec5e6792a452c7a76827c6b763"} Sep 06 13:59:40.894342 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:40.894305 2550 generic.go:356] "Generic (PLEG): container finished" podID="7be59bd6-458a-47f0-9e1f-4e22c97a8553" containerID="b1dc04079a69dc494890c55dd4ee1cd075982a316b0274da6934dc1f0e0e9b8f" exitCode=0 Sep 06 13:59:40.894718 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:40.894351 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" event={"ID":"7be59bd6-458a-47f0-9e1f-4e22c97a8553","Type":"ContainerDied","Data":"b1dc04079a69dc494890c55dd4ee1cd075982a316b0274da6934dc1f0e0e9b8f"} Sep 06 13:59:40.894755 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:40.894740 2550 scope.go:117] "RemoveContainer" containerID="b1dc04079a69dc494890c55dd4ee1cd075982a316b0274da6934dc1f0e0e9b8f" Sep 06 13:59:41.899496 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:41.899463 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7qqkq" event={"ID":"7be59bd6-458a-47f0-9e1f-4e22c97a8553","Type":"ContainerStarted","Data":"efb6b7e6b6bc4afb2b02b26a863f156e14526e1a22ab8c13d294d6e747a0f21f"} Sep 06 13:59:44.871022 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:44.870957 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7865949959-ltzq7" podUID="f1398bee-184b-4d0b-9056-ca03fbd96d4d" containerName="console" containerID="cri-o://65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23" gracePeriod=15 Sep 06 13:59:45.114278 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.114249 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:45.184310 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184276 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58fl5\" (UniqueName: \"kubernetes.io/projected/f1398bee-184b-4d0b-9056-ca03fbd96d4d-kube-api-access-58fl5\") pod \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " Sep 06 13:59:45.184468 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184355 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-serving-cert\") pod \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " Sep 06 13:59:45.184468 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184378 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-config\") pod \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " Sep 06 13:59:45.184468 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184409 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-service-ca\") pod \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " Sep 06 13:59:45.184468 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184435 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-oauth-serving-cert\") pod \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " Sep 06 13:59:45.184648 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184476 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-oauth-config\") pod \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\" (UID: \"f1398bee-184b-4d0b-9056-ca03fbd96d4d\") " Sep 06 13:59:45.184838 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184805 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-config" (OuterVolumeSpecName: "console-config") pod "f1398bee-184b-4d0b-9056-ca03fbd96d4d" (UID: "f1398bee-184b-4d0b-9056-ca03fbd96d4d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 13:59:45.184940 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184836 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1398bee-184b-4d0b-9056-ca03fbd96d4d" (UID: "f1398bee-184b-4d0b-9056-ca03fbd96d4d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 13:59:45.184940 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.184858 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1398bee-184b-4d0b-9056-ca03fbd96d4d" (UID: "f1398bee-184b-4d0b-9056-ca03fbd96d4d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 13:59:45.186752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.186725 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1398bee-184b-4d0b-9056-ca03fbd96d4d-kube-api-access-58fl5" (OuterVolumeSpecName: "kube-api-access-58fl5") pod "f1398bee-184b-4d0b-9056-ca03fbd96d4d" (UID: "f1398bee-184b-4d0b-9056-ca03fbd96d4d"). InnerVolumeSpecName "kube-api-access-58fl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 13:59:45.186752 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.186731 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1398bee-184b-4d0b-9056-ca03fbd96d4d" (UID: "f1398bee-184b-4d0b-9056-ca03fbd96d4d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 13:59:45.186862 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.186823 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1398bee-184b-4d0b-9056-ca03fbd96d4d" (UID: "f1398bee-184b-4d0b-9056-ca03fbd96d4d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 13:59:45.285955 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.285887 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:45.285955 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.285953 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-oauth-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:45.286165 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.285967 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-oauth-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:45.286165 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.285981 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58fl5\" (UniqueName: \"kubernetes.io/projected/f1398bee-184b-4d0b-9056-ca03fbd96d4d-kube-api-access-58fl5\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:45.286165 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.285994 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:45.286165 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.286005 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1398bee-184b-4d0b-9056-ca03fbd96d4d-console-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 13:59:45.914887 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.914852 2550 generic.go:356] "Generic (PLEG): container finished" podID="f1398bee-184b-4d0b-9056-ca03fbd96d4d" containerID="65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23" exitCode=0 Sep 06 13:59:45.915342 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.914916 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865949959-ltzq7" Sep 06 13:59:45.915342 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.914945 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865949959-ltzq7" event={"ID":"f1398bee-184b-4d0b-9056-ca03fbd96d4d","Type":"ContainerDied","Data":"65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23"} Sep 06 13:59:45.915342 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.915008 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865949959-ltzq7" event={"ID":"f1398bee-184b-4d0b-9056-ca03fbd96d4d","Type":"ContainerDied","Data":"4d3ea2d7a40ba82208eb3338f6abd4661c1d1a2aefbb824a2a92d35faf7d8fd2"} Sep 06 13:59:45.915342 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.915043 2550 scope.go:117] "RemoveContainer" containerID="65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23" Sep 06 13:59:45.923209 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.923190 2550 scope.go:117] "RemoveContainer" containerID="65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23" Sep 06 13:59:45.923550 ip-10-0-141-9 kubenswrapper[2550]: E0906 13:59:45.923529 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23\": container with ID starting with 65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23 not found: ID does not exist" containerID="65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23" Sep 06 13:59:45.923605 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.923565 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23"} err="failed to get container status \"65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23\": rpc error: code = NotFound desc = could not find container \"65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23\": container with ID starting with 65d3ba210881f0fa734f4669f658d052c72d18671ef0cc65346234fd6cb75a23 not found: ID does not exist" Sep 06 13:59:45.929347 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.929315 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7865949959-ltzq7"] Sep 06 13:59:45.933550 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:45.933525 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7865949959-ltzq7"] Sep 06 13:59:47.234001 ip-10-0-141-9 kubenswrapper[2550]: I0906 13:59:47.233966 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1398bee-184b-4d0b-9056-ca03fbd96d4d" path="/var/lib/kubelet/pods/f1398bee-184b-4d0b-9056-ca03fbd96d4d/volumes" Sep 06 14:00:17.206529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.206488 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55b764f646-wr9c5"] Sep 06 14:00:17.207204 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.207178 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1398bee-184b-4d0b-9056-ca03fbd96d4d" containerName="console" Sep 06 14:00:17.207287 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.207205 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1398bee-184b-4d0b-9056-ca03fbd96d4d" containerName="console" Sep 06 14:00:17.207287 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.207224 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0958c93-9d63-42d6-b109-ec5c7ae47828" containerName="registry" Sep 06 14:00:17.207287 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.207231 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0958c93-9d63-42d6-b109-ec5c7ae47828" containerName="registry" Sep 06 14:00:17.207435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.207310 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0958c93-9d63-42d6-b109-ec5c7ae47828" containerName="registry" Sep 06 14:00:17.207435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.207321 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1398bee-184b-4d0b-9056-ca03fbd96d4d" containerName="console" Sep 06 14:00:17.211683 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.211654 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.216477 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.216443 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b764f646-wr9c5"] Sep 06 14:00:17.335920 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.335852 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-oauth-config\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.336126 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.335932 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-serving-cert\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.336126 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.335961 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skp58\" (UniqueName: \"kubernetes.io/projected/841ca922-db20-4bc8-9d5f-a63f12054bbf-kube-api-access-skp58\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.336126 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.335987 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-oauth-serving-cert\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.336126 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.336086 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-config\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.336312 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.336137 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-service-ca\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.336312 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.336198 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-trusted-ca-bundle\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437394 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-serving-cert\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437394 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437399 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skp58\" (UniqueName: \"kubernetes.io/projected/841ca922-db20-4bc8-9d5f-a63f12054bbf-kube-api-access-skp58\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437436 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-oauth-serving-cert\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437470 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-config\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437500 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-service-ca\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437539 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-trusted-ca-bundle\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.437659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.437574 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-oauth-config\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.438388 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.438358 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-service-ca\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.438532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.438358 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-oauth-serving-cert\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.438532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.438434 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-config\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.438532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.438505 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-trusted-ca-bundle\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.440097 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.440073 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-serving-cert\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.440199 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.440175 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-oauth-config\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.446081 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.446047 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skp58\" (UniqueName: \"kubernetes.io/projected/841ca922-db20-4bc8-9d5f-a63f12054bbf-kube-api-access-skp58\") pod \"console-55b764f646-wr9c5\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.524154 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.524059 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:17.798880 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:17.798737 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b764f646-wr9c5"] Sep 06 14:00:17.801503 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:00:17.801478 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841ca922_db20_4bc8_9d5f_a63f12054bbf.slice/crio-3e40a76173bb94cfbb8d1b0b93d5033bc9f4daa47423251f92526daf4c60d495 WatchSource:0}: Error finding container 3e40a76173bb94cfbb8d1b0b93d5033bc9f4daa47423251f92526daf4c60d495: Status 404 returned error can't find the container with id 3e40a76173bb94cfbb8d1b0b93d5033bc9f4daa47423251f92526daf4c60d495 Sep 06 14:00:18.051996 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:18.051881 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b764f646-wr9c5" event={"ID":"841ca922-db20-4bc8-9d5f-a63f12054bbf","Type":"ContainerStarted","Data":"65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c"} Sep 06 14:00:18.051996 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:18.051959 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b764f646-wr9c5" event={"ID":"841ca922-db20-4bc8-9d5f-a63f12054bbf","Type":"ContainerStarted","Data":"3e40a76173bb94cfbb8d1b0b93d5033bc9f4daa47423251f92526daf4c60d495"} Sep 06 14:00:18.069021 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:18.068967 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55b764f646-wr9c5" podStartSLOduration=1.068949216 podStartE2EDuration="1.068949216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:00:18.067654094 +0000 UTC m=+167.331098821" watchObservedRunningTime="2026-09-06 14:00:18.068949216 +0000 UTC m=+167.332394041" Sep 06 14:00:27.525327 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:27.525285 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:27.525723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:27.525365 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:27.530415 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:27.530391 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:28.093064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:28.093032 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:00:28.139098 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:28.139065 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59db55c7db-lhffr"] Sep 06 14:00:53.164329 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.164260 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59db55c7db-lhffr" podUID="5528fcb6-b444-4c63-881c-d7f54ea7363d" containerName="console" containerID="cri-o://7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3" gracePeriod=15 Sep 06 14:00:53.409773 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.409739 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 14:00:53.495236 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495196 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-trusted-ca-bundle\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495236 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495235 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-oauth-config\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495510 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495295 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wjgw\" (UniqueName: \"kubernetes.io/projected/5528fcb6-b444-4c63-881c-d7f54ea7363d-kube-api-access-7wjgw\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495510 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495325 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-service-ca\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495510 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495342 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-config\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495503 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-oauth-serving-cert\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495555 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-serving-cert\") pod \"5528fcb6-b444-4c63-881c-d7f54ea7363d\" (UID: \"5528fcb6-b444-4c63-881c-d7f54ea7363d\") " Sep 06 14:00:53.495778 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495734 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:53.495778 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495754 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-service-ca" (OuterVolumeSpecName: "service-ca") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:53.495868 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.495740 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-config" (OuterVolumeSpecName: "console-config") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:53.496127 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.496100 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:00:53.497470 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.497450 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:53.497623 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.497599 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:00:53.497716 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.497694 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5528fcb6-b444-4c63-881c-d7f54ea7363d-kube-api-access-7wjgw" (OuterVolumeSpecName: "kube-api-access-7wjgw") pod "5528fcb6-b444-4c63-881c-d7f54ea7363d" (UID: "5528fcb6-b444-4c63-881c-d7f54ea7363d"). InnerVolumeSpecName "kube-api-access-7wjgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:00:53.596877 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596836 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:53.596877 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596867 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-trusted-ca-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:53.596877 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596878 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-oauth-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:53.596877 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596909 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjgw\" (UniqueName: \"kubernetes.io/projected/5528fcb6-b444-4c63-881c-d7f54ea7363d-kube-api-access-7wjgw\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:53.597140 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596919 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:53.597140 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596928 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-console-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:53.597140 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:53.596936 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5528fcb6-b444-4c63-881c-d7f54ea7363d-oauth-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:00:54.185987 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.185952 2550 generic.go:356] "Generic (PLEG): container finished" podID="5528fcb6-b444-4c63-881c-d7f54ea7363d" containerID="7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3" exitCode=0 Sep 06 14:00:54.186389 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.186002 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59db55c7db-lhffr" Sep 06 14:00:54.186389 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.186034 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59db55c7db-lhffr" event={"ID":"5528fcb6-b444-4c63-881c-d7f54ea7363d","Type":"ContainerDied","Data":"7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3"} Sep 06 14:00:54.186389 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.186093 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59db55c7db-lhffr" event={"ID":"5528fcb6-b444-4c63-881c-d7f54ea7363d","Type":"ContainerDied","Data":"9260ac3c2deadd0fd116de8a6c6a76e80d41a517990a5240e360b8bbd02c9b43"} Sep 06 14:00:54.186389 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.186121 2550 scope.go:117] "RemoveContainer" containerID="7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3" Sep 06 14:00:54.195464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.195442 2550 scope.go:117] "RemoveContainer" containerID="7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3" Sep 06 14:00:54.195788 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:00:54.195767 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3\": container with ID starting with 7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3 not found: ID does not exist" containerID="7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3" Sep 06 14:00:54.195854 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.195797 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3"} err="failed to get container status \"7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3\": rpc error: code = NotFound desc = could not find container \"7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3\": container with ID starting with 7eba68945af059e6d95aa563bb2eacac080470962670ead64a000ea145bdb3d3 not found: ID does not exist" Sep 06 14:00:54.205814 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.205739 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59db55c7db-lhffr"] Sep 06 14:00:54.208048 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:54.208015 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59db55c7db-lhffr"] Sep 06 14:00:55.234165 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:00:55.234125 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5528fcb6-b444-4c63-881c-d7f54ea7363d" path="/var/lib/kubelet/pods/5528fcb6-b444-4c63-881c-d7f54ea7363d/volumes" Sep 06 14:01:49.235193 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.235162 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7854d57d7b-mv5vh"] Sep 06 14:01:49.235701 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.235549 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5528fcb6-b444-4c63-881c-d7f54ea7363d" containerName="console" Sep 06 14:01:49.235701 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.235570 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5528fcb6-b444-4c63-881c-d7f54ea7363d" containerName="console" Sep 06 14:01:49.235701 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.235635 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5528fcb6-b444-4c63-881c-d7f54ea7363d" containerName="console" Sep 06 14:01:49.239902 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.239866 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.243838 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.243808 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7854d57d7b-mv5vh"] Sep 06 14:01:49.434313 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434271 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-trusted-ca-bundle\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.434313 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434315 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-serving-cert\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.434586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434345 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqr9\" (UniqueName: \"kubernetes.io/projected/961e1d56-71c4-4833-8db7-bf7ff3e92277-kube-api-access-fsqr9\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.434586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434422 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-oauth-serving-cert\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.434586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434502 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-service-ca\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.434586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434527 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-oauth-config\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.434586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.434558 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-config\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.535716 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.535613 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-config\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.535716 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.535657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-trusted-ca-bundle\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.535716 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.535676 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-serving-cert\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536011 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.535825 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqr9\" (UniqueName: \"kubernetes.io/projected/961e1d56-71c4-4833-8db7-bf7ff3e92277-kube-api-access-fsqr9\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536011 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.535886 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-oauth-serving-cert\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536011 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.535980 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-service-ca\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536011 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.536005 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-oauth-config\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.536520 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-config\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536666 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.536618 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-trusted-ca-bundle\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536726 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.536692 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-oauth-serving-cert\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.536726 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.536712 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-service-ca\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.538243 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.538213 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-oauth-config\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.538243 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.538220 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-serving-cert\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.543179 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.543155 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqr9\" (UniqueName: \"kubernetes.io/projected/961e1d56-71c4-4833-8db7-bf7ff3e92277-kube-api-access-fsqr9\") pod \"console-7854d57d7b-mv5vh\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.550123 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.550095 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:49.821909 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:49.821804 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7854d57d7b-mv5vh"] Sep 06 14:01:49.825142 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:01:49.825110 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961e1d56_71c4_4833_8db7_bf7ff3e92277.slice/crio-62d7a84f879a678d8733ed3da3a211f66e659022bf513c9a7dc8b3c36736494e WatchSource:0}: Error finding container 62d7a84f879a678d8733ed3da3a211f66e659022bf513c9a7dc8b3c36736494e: Status 404 returned error can't find the container with id 62d7a84f879a678d8733ed3da3a211f66e659022bf513c9a7dc8b3c36736494e Sep 06 14:01:50.397431 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:50.397392 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7854d57d7b-mv5vh" event={"ID":"961e1d56-71c4-4833-8db7-bf7ff3e92277","Type":"ContainerStarted","Data":"8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11"} Sep 06 14:01:50.397829 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:50.397443 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7854d57d7b-mv5vh" event={"ID":"961e1d56-71c4-4833-8db7-bf7ff3e92277","Type":"ContainerStarted","Data":"62d7a84f879a678d8733ed3da3a211f66e659022bf513c9a7dc8b3c36736494e"} Sep 06 14:01:50.413434 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:50.413377 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7854d57d7b-mv5vh" podStartSLOduration=1.413360326 podStartE2EDuration="1.413360326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:01:50.412248638 +0000 UTC m=+259.675693365" watchObservedRunningTime="2026-09-06 14:01:50.413360326 +0000 UTC m=+259.676805051" Sep 06 14:01:59.551206 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:59.551090 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:59.551206 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:59.551176 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:01:59.556114 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:01:59.556084 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:02:00.440669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:00.440637 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:02:00.480741 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:00.480707 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b764f646-wr9c5"] Sep 06 14:02:14.187323 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.187286 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w"] Sep 06 14:02:14.190980 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.190959 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.197084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.193658 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m7s6n\"" Sep 06 14:02:14.197084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.194010 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 06 14:02:14.197084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.194155 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 06 14:02:14.200155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.200123 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w"] Sep 06 14:02:14.221725 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.221686 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.221913 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.221743 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.221913 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.221855 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-kube-api-access-cbvfh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.323252 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.323209 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-kube-api-access-cbvfh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.323432 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.323275 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.323432 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.323320 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.323702 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.323680 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.323756 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.323738 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.331260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.331221 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-kube-api-access-cbvfh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.506182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.506086 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:14.775755 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:14.775651 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w"] Sep 06 14:02:14.778037 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:02:14.778004 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedfc8f5_a3dd_4f76_b3b9_506091fca1b4.slice/crio-9f4a087b97f1ff8f770effe606cb5c397695f321cf1fe64cb00fe80286719dc3 WatchSource:0}: Error finding container 9f4a087b97f1ff8f770effe606cb5c397695f321cf1fe64cb00fe80286719dc3: Status 404 returned error can't find the container with id 9f4a087b97f1ff8f770effe606cb5c397695f321cf1fe64cb00fe80286719dc3 Sep 06 14:02:15.498978 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:15.498931 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" event={"ID":"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4","Type":"ContainerStarted","Data":"9f4a087b97f1ff8f770effe606cb5c397695f321cf1fe64cb00fe80286719dc3"} Sep 06 14:02:21.526869 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:21.526829 2550 generic.go:356] "Generic (PLEG): container finished" podID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerID="1ac5fad16cf20d99780b8bef1d12f647f3562d35a57b09fb98c06ae2a85066da" exitCode=0 Sep 06 14:02:21.527265 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:21.526921 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" event={"ID":"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4","Type":"ContainerDied","Data":"1ac5fad16cf20d99780b8bef1d12f647f3562d35a57b09fb98c06ae2a85066da"} Sep 06 14:02:24.541178 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:24.541140 2550 generic.go:356] "Generic (PLEG): container finished" podID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerID="296d5a4cf73a3310a3ba83160525e14e2b5817f297a2a5b836ea71e4c893f73b" exitCode=0 Sep 06 14:02:24.541569 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:24.541235 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" event={"ID":"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4","Type":"ContainerDied","Data":"296d5a4cf73a3310a3ba83160525e14e2b5817f297a2a5b836ea71e4c893f73b"} Sep 06 14:02:25.505308 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.505261 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55b764f646-wr9c5" podUID="841ca922-db20-4bc8-9d5f-a63f12054bbf" containerName="console" containerID="cri-o://65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c" gracePeriod=15 Sep 06 14:02:25.765668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.765584 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:02:25.832698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832660 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-oauth-config\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.832910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832749 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-service-ca\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.832910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832778 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-trusted-ca-bundle\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.832910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832807 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-serving-cert\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.832910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832844 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-oauth-serving-cert\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.832910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832868 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skp58\" (UniqueName: \"kubernetes.io/projected/841ca922-db20-4bc8-9d5f-a63f12054bbf-kube-api-access-skp58\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.832910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.832904 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-config\") pod \"841ca922-db20-4bc8-9d5f-a63f12054bbf\" (UID: \"841ca922-db20-4bc8-9d5f-a63f12054bbf\") " Sep 06 14:02:25.833419 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.833375 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-service-ca" (OuterVolumeSpecName: "service-ca") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:02:25.833523 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.833447 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:02:25.833636 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.833616 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-config" (OuterVolumeSpecName: "console-config") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:02:25.833718 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.833674 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:02:25.835652 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.835609 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841ca922-db20-4bc8-9d5f-a63f12054bbf-kube-api-access-skp58" (OuterVolumeSpecName: "kube-api-access-skp58") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "kube-api-access-skp58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:02:25.835652 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.835634 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:02:25.835782 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.835662 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "841ca922-db20-4bc8-9d5f-a63f12054bbf" (UID: "841ca922-db20-4bc8-9d5f-a63f12054bbf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:02:25.934539 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934497 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:25.934539 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934537 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-trusted-ca-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:25.934750 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934549 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:25.934750 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934563 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-oauth-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:25.934750 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934575 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skp58\" (UniqueName: \"kubernetes.io/projected/841ca922-db20-4bc8-9d5f-a63f12054bbf-kube-api-access-skp58\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:25.934750 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934586 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:25.934750 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:25.934596 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841ca922-db20-4bc8-9d5f-a63f12054bbf-console-oauth-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:26.550741 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.550698 2550 generic.go:356] "Generic (PLEG): container finished" podID="841ca922-db20-4bc8-9d5f-a63f12054bbf" containerID="65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c" exitCode=0 Sep 06 14:02:26.550957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.550772 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b764f646-wr9c5" Sep 06 14:02:26.550957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.550799 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b764f646-wr9c5" event={"ID":"841ca922-db20-4bc8-9d5f-a63f12054bbf","Type":"ContainerDied","Data":"65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c"} Sep 06 14:02:26.550957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.550862 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b764f646-wr9c5" event={"ID":"841ca922-db20-4bc8-9d5f-a63f12054bbf","Type":"ContainerDied","Data":"3e40a76173bb94cfbb8d1b0b93d5033bc9f4daa47423251f92526daf4c60d495"} Sep 06 14:02:26.550957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.550907 2550 scope.go:117] "RemoveContainer" containerID="65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c" Sep 06 14:02:26.561074 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.561049 2550 scope.go:117] "RemoveContainer" containerID="65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c" Sep 06 14:02:26.561429 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:02:26.561408 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c\": container with ID starting with 65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c not found: ID does not exist" containerID="65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c" Sep 06 14:02:26.561509 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.561438 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c"} err="failed to get container status \"65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c\": rpc error: code = NotFound desc = could not find container \"65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c\": container with ID starting with 65e1734101697b9039cd895db34d72e130d8c4d11413afda4a55f29c84c5bf9c not found: ID does not exist" Sep 06 14:02:26.572157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.572119 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b764f646-wr9c5"] Sep 06 14:02:26.575420 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:26.575387 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55b764f646-wr9c5"] Sep 06 14:02:27.236445 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:27.236409 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841ca922-db20-4bc8-9d5f-a63f12054bbf" path="/var/lib/kubelet/pods/841ca922-db20-4bc8-9d5f-a63f12054bbf/volumes" Sep 06 14:02:31.163474 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:31.163383 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:02:31.163966 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:31.163859 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:02:31.174783 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:31.174754 2550 kubelet.go:1648] "Image garbage collection succeeded" Sep 06 14:02:31.575105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:31.575071 2550 generic.go:356] "Generic (PLEG): container finished" podID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerID="e59d5078e530dcf55eb72b852f6b032afbc25c7c1a7f20d4108db9a24cab4643" exitCode=0 Sep 06 14:02:31.577273 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:31.575154 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" event={"ID":"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4","Type":"ContainerDied","Data":"e59d5078e530dcf55eb72b852f6b032afbc25c7c1a7f20d4108db9a24cab4643"} Sep 06 14:02:32.703997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.703968 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:32.794342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.794301 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-bundle\") pod \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " Sep 06 14:02:32.794342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.794347 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-util\") pod \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " Sep 06 14:02:32.794665 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.794413 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-kube-api-access-cbvfh\") pod \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\" (UID: \"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4\") " Sep 06 14:02:32.795206 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.795173 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-bundle" (OuterVolumeSpecName: "bundle") pod "cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" (UID: "cedfc8f5-a3dd-4f76-b3b9-506091fca1b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:02:32.796688 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.796665 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-kube-api-access-cbvfh" (OuterVolumeSpecName: "kube-api-access-cbvfh") pod "cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" (UID: "cedfc8f5-a3dd-4f76-b3b9-506091fca1b4"). InnerVolumeSpecName "kube-api-access-cbvfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:02:32.799017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.798995 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-util" (OuterVolumeSpecName: "util") pod "cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" (UID: "cedfc8f5-a3dd-4f76-b3b9-506091fca1b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:02:32.895292 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.895198 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvfh\" (UniqueName: \"kubernetes.io/projected/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-kube-api-access-cbvfh\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:32.895292 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.895231 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:32.895292 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:32.895240 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cedfc8f5-a3dd-4f76-b3b9-506091fca1b4-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:33.585037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:33.584951 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" Sep 06 14:02:33.585037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:33.584971 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tb68w" event={"ID":"cedfc8f5-a3dd-4f76-b3b9-506091fca1b4","Type":"ContainerDied","Data":"9f4a087b97f1ff8f770effe606cb5c397695f321cf1fe64cb00fe80286719dc3"} Sep 06 14:02:33.585037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:33.585013 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4a087b97f1ff8f770effe606cb5c397695f321cf1fe64cb00fe80286719dc3" Sep 06 14:02:36.912705 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.912666 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6"] Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913002 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841ca922-db20-4bc8-9d5f-a63f12054bbf" containerName="console" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913015 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="841ca922-db20-4bc8-9d5f-a63f12054bbf" containerName="console" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913031 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="util" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913036 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="util" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913042 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="pull" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913048 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="pull" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913063 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="extract" Sep 06 14:02:36.913105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913068 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="extract" Sep 06 14:02:36.913342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913125 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cedfc8f5-a3dd-4f76-b3b9-506091fca1b4" containerName="extract" Sep 06 14:02:36.913342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.913131 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="841ca922-db20-4bc8-9d5f-a63f12054bbf" containerName="console" Sep 06 14:02:36.917199 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.917176 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:36.919872 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.919849 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 06 14:02:36.920013 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.919878 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 06 14:02:36.920013 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.919989 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-66fm9\"" Sep 06 14:02:36.928408 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:36.928378 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6"] Sep 06 14:02:37.029034 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.028997 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/082e6a94-454f-414d-900a-75ca0ed346ee-kube-api-access-gjp6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-mx4m6\" (UID: \"082e6a94-454f-414d-900a-75ca0ed346ee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.029206 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.029092 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/082e6a94-454f-414d-900a-75ca0ed346ee-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mx4m6\" (UID: \"082e6a94-454f-414d-900a-75ca0ed346ee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.130347 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.130306 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/082e6a94-454f-414d-900a-75ca0ed346ee-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mx4m6\" (UID: \"082e6a94-454f-414d-900a-75ca0ed346ee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.130347 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.130352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/082e6a94-454f-414d-900a-75ca0ed346ee-kube-api-access-gjp6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-mx4m6\" (UID: \"082e6a94-454f-414d-900a-75ca0ed346ee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.130706 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.130686 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/082e6a94-454f-414d-900a-75ca0ed346ee-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mx4m6\" (UID: \"082e6a94-454f-414d-900a-75ca0ed346ee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.139594 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.139557 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/082e6a94-454f-414d-900a-75ca0ed346ee-kube-api-access-gjp6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-mx4m6\" (UID: \"082e6a94-454f-414d-900a-75ca0ed346ee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.227585 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.227541 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" Sep 06 14:02:37.511520 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.511436 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6"] Sep 06 14:02:37.514924 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:02:37.514878 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082e6a94_454f_414d_900a_75ca0ed346ee.slice/crio-965a716c2fb57a3a2ca57c5535e7fec3b1f352702d5bad63e6f509001438d43d WatchSource:0}: Error finding container 965a716c2fb57a3a2ca57c5535e7fec3b1f352702d5bad63e6f509001438d43d: Status 404 returned error can't find the container with id 965a716c2fb57a3a2ca57c5535e7fec3b1f352702d5bad63e6f509001438d43d Sep 06 14:02:37.517609 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.517591 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:02:37.602816 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:37.602783 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" event={"ID":"082e6a94-454f-414d-900a-75ca0ed346ee","Type":"ContainerStarted","Data":"965a716c2fb57a3a2ca57c5535e7fec3b1f352702d5bad63e6f509001438d43d"} Sep 06 14:02:39.616464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:39.616422 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" event={"ID":"082e6a94-454f-414d-900a-75ca0ed346ee","Type":"ContainerStarted","Data":"09339d2319e5d6bf7b0a1db0dd221c9709f62c03350b9ac7e4452e4a6e29e73e"} Sep 06 14:02:39.635239 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:39.635181 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mx4m6" podStartSLOduration=2.167829415 podStartE2EDuration="3.635164047s" totalImagesPullingTime="1.467334632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:02:39.632976985 +0000 UTC m=+308.896421706" watchObservedRunningTime="2026-09-06 14:02:39.635164047 +0000 UTC m=+308.898608770" Sep 06 14:02:41.360602 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.360564 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4"] Sep 06 14:02:41.364100 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.364071 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.366421 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.366397 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 06 14:02:41.367033 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.367018 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m7s6n\"" Sep 06 14:02:41.367033 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.367026 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 06 14:02:41.370221 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.370195 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4"] Sep 06 14:02:41.467018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.466977 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.467232 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.467043 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l299g\" (UniqueName: \"kubernetes.io/projected/dcd78a53-8420-4af6-b176-9ac11bff581a-kube-api-access-l299g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.467232 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.467134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.568464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.568418 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l299g\" (UniqueName: \"kubernetes.io/projected/dcd78a53-8420-4af6-b176-9ac11bff581a-kube-api-access-l299g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.568464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.568467 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.568724 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.568648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.568809 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.568791 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.568973 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.568956 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.576393 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.576363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l299g\" (UniqueName: \"kubernetes.io/projected/dcd78a53-8420-4af6-b176-9ac11bff581a-kube-api-access-l299g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.673799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.673761 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:41.949055 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:41.948943 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4"] Sep 06 14:02:41.952231 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:02:41.952199 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd78a53_8420_4af6_b176_9ac11bff581a.slice/crio-8d2612551f9142c414d8cf1fc011c42885a9820533678f1adaa954a2d9886638 WatchSource:0}: Error finding container 8d2612551f9142c414d8cf1fc011c42885a9820533678f1adaa954a2d9886638: Status 404 returned error can't find the container with id 8d2612551f9142c414d8cf1fc011c42885a9820533678f1adaa954a2d9886638 Sep 06 14:02:42.386389 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.386305 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng"] Sep 06 14:02:42.389399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.389379 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.391612 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.391581 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vgr44\"" Sep 06 14:02:42.391743 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.391626 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 06 14:02:42.392316 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.392300 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 06 14:02:42.398129 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.398105 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng"] Sep 06 14:02:42.475448 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.475413 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad0239d-fc3d-4d84-844a-0570bcd488fd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gm7ng\" (UID: \"7ad0239d-fc3d-4d84-844a-0570bcd488fd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.475448 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.475450 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkvb\" (UniqueName: \"kubernetes.io/projected/7ad0239d-fc3d-4d84-844a-0570bcd488fd-kube-api-access-8lkvb\") pod \"cert-manager-cainjector-559cdd8c87-gm7ng\" (UID: \"7ad0239d-fc3d-4d84-844a-0570bcd488fd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.576482 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.576444 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad0239d-fc3d-4d84-844a-0570bcd488fd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gm7ng\" (UID: \"7ad0239d-fc3d-4d84-844a-0570bcd488fd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.576482 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.576486 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkvb\" (UniqueName: \"kubernetes.io/projected/7ad0239d-fc3d-4d84-844a-0570bcd488fd-kube-api-access-8lkvb\") pod \"cert-manager-cainjector-559cdd8c87-gm7ng\" (UID: \"7ad0239d-fc3d-4d84-844a-0570bcd488fd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.584157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.584124 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad0239d-fc3d-4d84-844a-0570bcd488fd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gm7ng\" (UID: \"7ad0239d-fc3d-4d84-844a-0570bcd488fd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.584309 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.584292 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkvb\" (UniqueName: \"kubernetes.io/projected/7ad0239d-fc3d-4d84-844a-0570bcd488fd-kube-api-access-8lkvb\") pod \"cert-manager-cainjector-559cdd8c87-gm7ng\" (UID: \"7ad0239d-fc3d-4d84-844a-0570bcd488fd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.631425 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.631390 2550 generic.go:356] "Generic (PLEG): container finished" podID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerID="0e387ca6560a1be41d1440d4420e65e2d591716d863b08ff32132ceb6cd29de1" exitCode=0 Sep 06 14:02:42.631605 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.631484 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" event={"ID":"dcd78a53-8420-4af6-b176-9ac11bff581a","Type":"ContainerDied","Data":"0e387ca6560a1be41d1440d4420e65e2d591716d863b08ff32132ceb6cd29de1"} Sep 06 14:02:42.631605 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.631542 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" event={"ID":"dcd78a53-8420-4af6-b176-9ac11bff581a","Type":"ContainerStarted","Data":"8d2612551f9142c414d8cf1fc011c42885a9820533678f1adaa954a2d9886638"} Sep 06 14:02:42.710513 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.710472 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" Sep 06 14:02:42.979285 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:42.979246 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng"] Sep 06 14:02:42.981962 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:02:42.981930 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad0239d_fc3d_4d84_844a_0570bcd488fd.slice/crio-808738041ef617ea65d3efc3a4ba946e88c09996209f2505a79ffadaa042ae02 WatchSource:0}: Error finding container 808738041ef617ea65d3efc3a4ba946e88c09996209f2505a79ffadaa042ae02: Status 404 returned error can't find the container with id 808738041ef617ea65d3efc3a4ba946e88c09996209f2505a79ffadaa042ae02 Sep 06 14:02:43.639077 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:43.639041 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" event={"ID":"7ad0239d-fc3d-4d84-844a-0570bcd488fd","Type":"ContainerStarted","Data":"808738041ef617ea65d3efc3a4ba946e88c09996209f2505a79ffadaa042ae02"} Sep 06 14:02:46.658258 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:46.658215 2550 generic.go:356] "Generic (PLEG): container finished" podID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerID="2bd462a29c206ec87d2550469cc81ec49dd02489260ca3ed58a4c8d695e9b8d3" exitCode=0 Sep 06 14:02:46.658713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:46.658302 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" event={"ID":"dcd78a53-8420-4af6-b176-9ac11bff581a","Type":"ContainerDied","Data":"2bd462a29c206ec87d2550469cc81ec49dd02489260ca3ed58a4c8d695e9b8d3"} Sep 06 14:02:46.660165 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:46.660137 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" event={"ID":"7ad0239d-fc3d-4d84-844a-0570bcd488fd","Type":"ContainerStarted","Data":"7946cf188e6bf9dd61946ff2cee8a86091f3cac832f52b81249ab82ff1a9f750"} Sep 06 14:02:46.685485 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:46.685431 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gm7ng" podStartSLOduration=1.765474494 podStartE2EDuration="4.685413047s" totalImagesPullingTime="2.919938553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:02:46.684546283 +0000 UTC m=+315.947991032" watchObservedRunningTime="2026-09-06 14:02:46.685413047 +0000 UTC m=+315.948857771" Sep 06 14:02:47.666802 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:47.666771 2550 generic.go:356] "Generic (PLEG): container finished" podID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerID="be577b579238a0ac7e78397075c1836f476483e78507c9c025a005f1de910e45" exitCode=0 Sep 06 14:02:47.667182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:47.666849 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" event={"ID":"dcd78a53-8420-4af6-b176-9ac11bff581a","Type":"ContainerDied","Data":"be577b579238a0ac7e78397075c1836f476483e78507c9c025a005f1de910e45"} Sep 06 14:02:48.804033 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.804003 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:48.935535 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.935442 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l299g\" (UniqueName: \"kubernetes.io/projected/dcd78a53-8420-4af6-b176-9ac11bff581a-kube-api-access-l299g\") pod \"dcd78a53-8420-4af6-b176-9ac11bff581a\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " Sep 06 14:02:48.935535 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.935500 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-bundle\") pod \"dcd78a53-8420-4af6-b176-9ac11bff581a\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " Sep 06 14:02:48.935720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.935558 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-util\") pod \"dcd78a53-8420-4af6-b176-9ac11bff581a\" (UID: \"dcd78a53-8420-4af6-b176-9ac11bff581a\") " Sep 06 14:02:48.936130 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.936085 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-bundle" (OuterVolumeSpecName: "bundle") pod "dcd78a53-8420-4af6-b176-9ac11bff581a" (UID: "dcd78a53-8420-4af6-b176-9ac11bff581a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:02:48.937622 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.937593 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd78a53-8420-4af6-b176-9ac11bff581a-kube-api-access-l299g" (OuterVolumeSpecName: "kube-api-access-l299g") pod "dcd78a53-8420-4af6-b176-9ac11bff581a" (UID: "dcd78a53-8420-4af6-b176-9ac11bff581a"). InnerVolumeSpecName "kube-api-access-l299g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:02:48.940405 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:48.940374 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-util" (OuterVolumeSpecName: "util") pod "dcd78a53-8420-4af6-b176-9ac11bff581a" (UID: "dcd78a53-8420-4af6-b176-9ac11bff581a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:02:49.036441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:49.036392 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:49.036441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:49.036437 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l299g\" (UniqueName: \"kubernetes.io/projected/dcd78a53-8420-4af6-b176-9ac11bff581a-kube-api-access-l299g\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:49.036441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:49.036451 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcd78a53-8420-4af6-b176-9ac11bff581a-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:02:49.676875 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:49.676836 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" event={"ID":"dcd78a53-8420-4af6-b176-9ac11bff581a","Type":"ContainerDied","Data":"8d2612551f9142c414d8cf1fc011c42885a9820533678f1adaa954a2d9886638"} Sep 06 14:02:49.677067 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:49.676885 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2612551f9142c414d8cf1fc011c42885a9820533678f1adaa954a2d9886638" Sep 06 14:02:49.677067 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:49.676906 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fq84n4" Sep 06 14:02:53.704103 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704068 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l"] Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704337 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="util" Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704347 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="util" Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704368 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="extract" Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704373 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="extract" Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704382 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="pull" Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704387 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="pull" Sep 06 14:02:53.704491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.704436 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd78a53-8420-4af6-b176-9ac11bff581a" containerName="extract" Sep 06 14:02:53.707271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.707249 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:53.709469 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.709448 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 06 14:02:53.710310 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.710292 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jqw5f\"" Sep 06 14:02:53.710395 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.710298 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 06 14:02:53.714235 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.714215 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l"] Sep 06 14:02:53.772274 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.772234 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnx8\" (UniqueName: \"kubernetes.io/projected/84c55162-9491-42d8-8a62-9a47673c3c5c-kube-api-access-7wnx8\") pod \"openshift-lws-operator-fd8ccff4c-b6g5l\" (UID: \"84c55162-9491-42d8-8a62-9a47673c3c5c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:53.772274 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.772275 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84c55162-9491-42d8-8a62-9a47673c3c5c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b6g5l\" (UID: \"84c55162-9491-42d8-8a62-9a47673c3c5c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:53.872667 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.872632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnx8\" (UniqueName: \"kubernetes.io/projected/84c55162-9491-42d8-8a62-9a47673c3c5c-kube-api-access-7wnx8\") pod \"openshift-lws-operator-fd8ccff4c-b6g5l\" (UID: \"84c55162-9491-42d8-8a62-9a47673c3c5c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:53.872667 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.872672 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84c55162-9491-42d8-8a62-9a47673c3c5c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b6g5l\" (UID: \"84c55162-9491-42d8-8a62-9a47673c3c5c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:53.873074 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.873055 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84c55162-9491-42d8-8a62-9a47673c3c5c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b6g5l\" (UID: \"84c55162-9491-42d8-8a62-9a47673c3c5c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:53.879670 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:53.879641 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnx8\" (UniqueName: \"kubernetes.io/projected/84c55162-9491-42d8-8a62-9a47673c3c5c-kube-api-access-7wnx8\") pod \"openshift-lws-operator-fd8ccff4c-b6g5l\" (UID: \"84c55162-9491-42d8-8a62-9a47673c3c5c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:54.016839 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:54.016737 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" Sep 06 14:02:54.286202 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:54.286177 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l"] Sep 06 14:02:54.288770 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:02:54.288739 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84c55162_9491_42d8_8a62_9a47673c3c5c.slice/crio-e169b5b0a67b8224f5d3a8f7e1580f41618340090c1fcf032714cc2262dc42a6 WatchSource:0}: Error finding container e169b5b0a67b8224f5d3a8f7e1580f41618340090c1fcf032714cc2262dc42a6: Status 404 returned error can't find the container with id e169b5b0a67b8224f5d3a8f7e1580f41618340090c1fcf032714cc2262dc42a6 Sep 06 14:02:54.706777 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:54.706738 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" event={"ID":"84c55162-9491-42d8-8a62-9a47673c3c5c","Type":"ContainerStarted","Data":"e169b5b0a67b8224f5d3a8f7e1580f41618340090c1fcf032714cc2262dc42a6"} Sep 06 14:02:56.716876 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:56.716842 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" event={"ID":"84c55162-9491-42d8-8a62-9a47673c3c5c","Type":"ContainerStarted","Data":"43cd3d06ff6438f541a92d13909843f33c5a0552638e44bebbb2c42712c1cf92"} Sep 06 14:02:56.731155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:56.731095 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b6g5l" podStartSLOduration=1.686620727 podStartE2EDuration="3.731078605s" totalImagesPullingTime="2.044457878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:02:56.730177285 +0000 UTC m=+325.993622011" watchObservedRunningTime="2026-09-06 14:02:56.731078605 +0000 UTC m=+325.994523329" Sep 06 14:02:59.413864 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.413823 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv"] Sep 06 14:02:59.419451 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.419417 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.421846 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.421817 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 06 14:02:59.422013 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.421924 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m7s6n\"" Sep 06 14:02:59.422079 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.422042 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 06 14:02:59.423310 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.423287 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv"] Sep 06 14:02:59.509139 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.509107 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.509318 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.509155 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcvk\" (UniqueName: \"kubernetes.io/projected/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-kube-api-access-rrcvk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.509318 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.509226 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.609698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.609660 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.609929 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.609712 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcvk\" (UniqueName: \"kubernetes.io/projected/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-kube-api-access-rrcvk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.609929 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.609769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.610177 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.610154 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.610249 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.610162 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.617433 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.617405 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcvk\" (UniqueName: \"kubernetes.io/projected/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-kube-api-access-rrcvk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.729827 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.729794 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:02:59.999724 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:02:59.999695 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv"] Sep 06 14:03:00.002203 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:00.002169 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5991e434_eb83_48f1_8c99_b0a72bfb1ce0.slice/crio-2e1b5fa31954413e4224510f36715eb26995972cd1420697beedf12a04d3be9a WatchSource:0}: Error finding container 2e1b5fa31954413e4224510f36715eb26995972cd1420697beedf12a04d3be9a: Status 404 returned error can't find the container with id 2e1b5fa31954413e4224510f36715eb26995972cd1420697beedf12a04d3be9a Sep 06 14:03:00.736532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:00.736491 2550 generic.go:356] "Generic (PLEG): container finished" podID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerID="229a72a25ffe3a951f50a95984959247565e0cdb2ffcf3d085210add7b0f02fb" exitCode=0 Sep 06 14:03:00.737043 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:00.736539 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" event={"ID":"5991e434-eb83-48f1-8c99-b0a72bfb1ce0","Type":"ContainerDied","Data":"229a72a25ffe3a951f50a95984959247565e0cdb2ffcf3d085210add7b0f02fb"} Sep 06 14:03:00.737043 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:00.736581 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" event={"ID":"5991e434-eb83-48f1-8c99-b0a72bfb1ce0","Type":"ContainerStarted","Data":"2e1b5fa31954413e4224510f36715eb26995972cd1420697beedf12a04d3be9a"} Sep 06 14:03:01.745297 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:01.745258 2550 generic.go:356] "Generic (PLEG): container finished" podID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerID="d85c848c5e499cbbb820804d6c787a298c24ae07c581b4a6de79d490d8d0aac6" exitCode=0 Sep 06 14:03:01.745795 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:01.745324 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" event={"ID":"5991e434-eb83-48f1-8c99-b0a72bfb1ce0","Type":"ContainerDied","Data":"d85c848c5e499cbbb820804d6c787a298c24ae07c581b4a6de79d490d8d0aac6"} Sep 06 14:03:02.752470 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:02.752439 2550 generic.go:356] "Generic (PLEG): container finished" podID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerID="16a7242bc24f4e2a3b3bebbcf955a78e84d0111dc14fb8753c0776e1f560ee6d" exitCode=0 Sep 06 14:03:02.752863 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:02.752488 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" event={"ID":"5991e434-eb83-48f1-8c99-b0a72bfb1ce0","Type":"ContainerDied","Data":"16a7242bc24f4e2a3b3bebbcf955a78e84d0111dc14fb8753c0776e1f560ee6d"} Sep 06 14:03:03.896433 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:03.896399 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:03:03.935964 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:03.935930 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-util\") pod \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " Sep 06 14:03:03.935964 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:03.935967 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-bundle\") pod \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " Sep 06 14:03:03.936723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:03.936693 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-bundle" (OuterVolumeSpecName: "bundle") pod "5991e434-eb83-48f1-8c99-b0a72bfb1ce0" (UID: "5991e434-eb83-48f1-8c99-b0a72bfb1ce0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:03:03.941453 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:03.941417 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-util" (OuterVolumeSpecName: "util") pod "5991e434-eb83-48f1-8c99-b0a72bfb1ce0" (UID: "5991e434-eb83-48f1-8c99-b0a72bfb1ce0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:03:04.036435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.036347 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrcvk\" (UniqueName: \"kubernetes.io/projected/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-kube-api-access-rrcvk\") pod \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\" (UID: \"5991e434-eb83-48f1-8c99-b0a72bfb1ce0\") " Sep 06 14:03:04.036584 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.036523 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:04.036584 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.036541 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:04.038464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.038439 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-kube-api-access-rrcvk" (OuterVolumeSpecName: "kube-api-access-rrcvk") pod "5991e434-eb83-48f1-8c99-b0a72bfb1ce0" (UID: "5991e434-eb83-48f1-8c99-b0a72bfb1ce0"). InnerVolumeSpecName "kube-api-access-rrcvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:03:04.137006 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.136963 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcvk\" (UniqueName: \"kubernetes.io/projected/5991e434-eb83-48f1-8c99-b0a72bfb1ce0-kube-api-access-rrcvk\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:04.763176 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.763148 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" Sep 06 14:03:04.763176 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.763157 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff7twv" event={"ID":"5991e434-eb83-48f1-8c99-b0a72bfb1ce0","Type":"ContainerDied","Data":"2e1b5fa31954413e4224510f36715eb26995972cd1420697beedf12a04d3be9a"} Sep 06 14:03:04.763400 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:04.763203 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1b5fa31954413e4224510f36715eb26995972cd1420697beedf12a04d3be9a" Sep 06 14:03:15.591137 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591094 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt"] Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591517 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="util" Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591535 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="util" Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591549 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="pull" Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591557 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="pull" Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591589 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="extract" Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591597 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="extract" Sep 06 14:03:15.591715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.591681 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5991e434-eb83-48f1-8c99-b0a72bfb1ce0" containerName="extract" Sep 06 14:03:15.598048 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.598019 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.601381 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.601351 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 06 14:03:15.601550 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.601351 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 06 14:03:15.601550 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.601458 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m7s6n\"" Sep 06 14:03:15.602405 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.602382 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt"] Sep 06 14:03:15.613206 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.613114 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.613386 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.613294 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zc8j\" (UniqueName: \"kubernetes.io/projected/c1959aac-12a6-4ace-914d-569c443baf66-kube-api-access-4zc8j\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.613386 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.613368 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.713781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.713734 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zc8j\" (UniqueName: \"kubernetes.io/projected/c1959aac-12a6-4ace-914d-569c443baf66-kube-api-access-4zc8j\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.714018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.713801 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.714018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.713837 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.714207 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.714189 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.714260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.714223 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.721773 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.721736 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zc8j\" (UniqueName: \"kubernetes.io/projected/c1959aac-12a6-4ace-914d-569c443baf66-kube-api-access-4zc8j\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:15.908703 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:15.908610 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:16.181777 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.181745 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt"] Sep 06 14:03:16.184207 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:16.184180 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1959aac_12a6_4ace_914d_569c443baf66.slice/crio-699ab9c46947c2dcddff54432f05d4f75b288b7285d24a3fc21e7bc8babc5211 WatchSource:0}: Error finding container 699ab9c46947c2dcddff54432f05d4f75b288b7285d24a3fc21e7bc8babc5211: Status 404 returned error can't find the container with id 699ab9c46947c2dcddff54432f05d4f75b288b7285d24a3fc21e7bc8babc5211 Sep 06 14:03:16.730846 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.730814 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj"] Sep 06 14:03:16.739270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.739240 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:16.742805 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.742771 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 06 14:03:16.742970 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.742805 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 06 14:03:16.742970 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.742885 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 06 14:03:16.743291 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.743275 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 06 14:03:16.743422 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.743405 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-c7zg9\"" Sep 06 14:03:16.756926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.756878 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj"] Sep 06 14:03:16.816616 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.816576 2550 generic.go:356] "Generic (PLEG): container finished" podID="c1959aac-12a6-4ace-914d-569c443baf66" containerID="4d9a618a18ca95f6087b0dcc7a68193e17aa16f929d04345e5b83f220dcb0b29" exitCode=0 Sep 06 14:03:16.816804 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.816664 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" event={"ID":"c1959aac-12a6-4ace-914d-569c443baf66","Type":"ContainerDied","Data":"4d9a618a18ca95f6087b0dcc7a68193e17aa16f929d04345e5b83f220dcb0b29"} Sep 06 14:03:16.816804 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.816719 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" event={"ID":"c1959aac-12a6-4ace-914d-569c443baf66","Type":"ContainerStarted","Data":"699ab9c46947c2dcddff54432f05d4f75b288b7285d24a3fc21e7bc8babc5211"} Sep 06 14:03:16.922471 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.922423 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dce402da-d573-4c7a-afb5-e79852b0dc02-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:16.922471 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.922473 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnww\" (UniqueName: \"kubernetes.io/projected/dce402da-d573-4c7a-afb5-e79852b0dc02-kube-api-access-xpnww\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:16.922720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.922526 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dce402da-d573-4c7a-afb5-e79852b0dc02-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:16.922720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.922575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dce402da-d573-4c7a-afb5-e79852b0dc02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:16.922720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:16.922605 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dce402da-d573-4c7a-afb5-e79852b0dc02-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.023306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.023216 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dce402da-d573-4c7a-afb5-e79852b0dc02-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.023306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.023259 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnww\" (UniqueName: \"kubernetes.io/projected/dce402da-d573-4c7a-afb5-e79852b0dc02-kube-api-access-xpnww\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.023306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.023286 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dce402da-d573-4c7a-afb5-e79852b0dc02-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.023306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.023309 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dce402da-d573-4c7a-afb5-e79852b0dc02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.023645 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.023328 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dce402da-d573-4c7a-afb5-e79852b0dc02-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.023788 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.023750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dce402da-d573-4c7a-afb5-e79852b0dc02-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.025875 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.025850 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dce402da-d573-4c7a-afb5-e79852b0dc02-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.026089 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.026069 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dce402da-d573-4c7a-afb5-e79852b0dc02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.026141 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.026069 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dce402da-d573-4c7a-afb5-e79852b0dc02-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.035390 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.035348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnww\" (UniqueName: \"kubernetes.io/projected/dce402da-d573-4c7a-afb5-e79852b0dc02-kube-api-access-xpnww\") pod \"opendatahub-operator-controller-manager-7b6858f974-vhprj\" (UID: \"dce402da-d573-4c7a-afb5-e79852b0dc02\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.050242 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.050204 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:17.345205 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.345176 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj"] Sep 06 14:03:17.347764 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:17.347739 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce402da_d573_4c7a_afb5_e79852b0dc02.slice/crio-7df6d08f0b70acc83db3a266e73057fc0b627481e55514d340efa4d7f1731a24 WatchSource:0}: Error finding container 7df6d08f0b70acc83db3a266e73057fc0b627481e55514d340efa4d7f1731a24: Status 404 returned error can't find the container with id 7df6d08f0b70acc83db3a266e73057fc0b627481e55514d340efa4d7f1731a24 Sep 06 14:03:17.822393 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:17.822357 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" event={"ID":"dce402da-d573-4c7a-afb5-e79852b0dc02","Type":"ContainerStarted","Data":"7df6d08f0b70acc83db3a266e73057fc0b627481e55514d340efa4d7f1731a24"} Sep 06 14:03:18.832911 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:18.832855 2550 generic.go:356] "Generic (PLEG): container finished" podID="c1959aac-12a6-4ace-914d-569c443baf66" containerID="835f5f939f5e39af6676d721c0cc4bb84eafbf016d4262a4347f55f6317b9088" exitCode=0 Sep 06 14:03:18.833379 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:18.832936 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" event={"ID":"c1959aac-12a6-4ace-914d-569c443baf66","Type":"ContainerDied","Data":"835f5f939f5e39af6676d721c0cc4bb84eafbf016d4262a4347f55f6317b9088"} Sep 06 14:03:20.847623 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:20.847585 2550 generic.go:356] "Generic (PLEG): container finished" podID="c1959aac-12a6-4ace-914d-569c443baf66" containerID="618331c7f9aa4d180af52ac2312498118347c8f6f08fd105c4bd86d1f690b349" exitCode=0 Sep 06 14:03:20.848029 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:20.847658 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" event={"ID":"c1959aac-12a6-4ace-914d-569c443baf66","Type":"ContainerDied","Data":"618331c7f9aa4d180af52ac2312498118347c8f6f08fd105c4bd86d1f690b349"} Sep 06 14:03:20.849310 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:20.849281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" event={"ID":"dce402da-d573-4c7a-afb5-e79852b0dc02","Type":"ContainerStarted","Data":"16eac834185ae808308307bcc4547ed1989ebf9f57b59b66b7abd3244c62cc71"} Sep 06 14:03:21.854913 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:21.854801 2550 generic.go:356] "Generic (PLEG): container finished" podID="dce402da-d573-4c7a-afb5-e79852b0dc02" containerID="16eac834185ae808308307bcc4547ed1989ebf9f57b59b66b7abd3244c62cc71" exitCode=0 Sep 06 14:03:21.855309 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:21.854910 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" event={"ID":"dce402da-d573-4c7a-afb5-e79852b0dc02","Type":"ContainerDied","Data":"16eac834185ae808308307bcc4547ed1989ebf9f57b59b66b7abd3244c62cc71"} Sep 06 14:03:21.984025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:21.983995 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:22.067613 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.067579 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-bundle\") pod \"c1959aac-12a6-4ace-914d-569c443baf66\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " Sep 06 14:03:22.067807 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.067633 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zc8j\" (UniqueName: \"kubernetes.io/projected/c1959aac-12a6-4ace-914d-569c443baf66-kube-api-access-4zc8j\") pod \"c1959aac-12a6-4ace-914d-569c443baf66\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " Sep 06 14:03:22.067807 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.067676 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-util\") pod \"c1959aac-12a6-4ace-914d-569c443baf66\" (UID: \"c1959aac-12a6-4ace-914d-569c443baf66\") " Sep 06 14:03:22.068504 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.068478 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-bundle" (OuterVolumeSpecName: "bundle") pod "c1959aac-12a6-4ace-914d-569c443baf66" (UID: "c1959aac-12a6-4ace-914d-569c443baf66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:03:22.069835 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.069812 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1959aac-12a6-4ace-914d-569c443baf66-kube-api-access-4zc8j" (OuterVolumeSpecName: "kube-api-access-4zc8j") pod "c1959aac-12a6-4ace-914d-569c443baf66" (UID: "c1959aac-12a6-4ace-914d-569c443baf66"). InnerVolumeSpecName "kube-api-access-4zc8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:03:22.073293 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.073268 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-util" (OuterVolumeSpecName: "util") pod "c1959aac-12a6-4ace-914d-569c443baf66" (UID: "c1959aac-12a6-4ace-914d-569c443baf66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:03:22.168704 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.168669 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zc8j\" (UniqueName: \"kubernetes.io/projected/c1959aac-12a6-4ace-914d-569c443baf66-kube-api-access-4zc8j\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:22.168704 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.168704 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:22.168936 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.168713 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1959aac-12a6-4ace-914d-569c443baf66-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:22.861745 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.861716 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" Sep 06 14:03:22.862320 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.861717 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hxftt" event={"ID":"c1959aac-12a6-4ace-914d-569c443baf66","Type":"ContainerDied","Data":"699ab9c46947c2dcddff54432f05d4f75b288b7285d24a3fc21e7bc8babc5211"} Sep 06 14:03:22.862320 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.861853 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699ab9c46947c2dcddff54432f05d4f75b288b7285d24a3fc21e7bc8babc5211" Sep 06 14:03:22.864038 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.864007 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" event={"ID":"dce402da-d573-4c7a-afb5-e79852b0dc02","Type":"ContainerStarted","Data":"e4d5276a7ed0666a5bc41e7bc9fa4b8c4a84295b88c9e8d4df665ac1803d0006"} Sep 06 14:03:22.864143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.864120 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:22.885018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:22.884966 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" podStartSLOduration=2.363241928 podStartE2EDuration="6.884926291s" totalImagesPullingTime="3.392095257s" totalInitContainerRuntime="1.129589106s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:22.883222866 +0000 UTC m=+352.146667616" watchObservedRunningTime="2026-09-06 14:03:22.884926291 +0000 UTC m=+352.148371086" Sep 06 14:03:33.871457 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:33.871422 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-vhprj" Sep 06 14:03:35.826515 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.826478 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj"] Sep 06 14:03:35.827025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.826966 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="util" Sep 06 14:03:35.827025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.826986 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="util" Sep 06 14:03:35.827025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.827003 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="extract" Sep 06 14:03:35.827025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.827011 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="extract" Sep 06 14:03:35.827025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.827021 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="pull" Sep 06 14:03:35.827273 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.827029 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="pull" Sep 06 14:03:35.827273 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.827112 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1959aac-12a6-4ace-914d-569c443baf66" containerName="extract" Sep 06 14:03:35.836286 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.836248 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.839088 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.839054 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 06 14:03:35.839088 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.839061 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 06 14:03:35.839289 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.839163 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8wpml\"" Sep 06 14:03:35.850142 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.850110 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj"] Sep 06 14:03:35.885270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.885228 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2bdce0-1beb-425b-a752-d49b305fd470-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.885270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.885274 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxxw\" (UniqueName: \"kubernetes.io/projected/8c2bdce0-1beb-425b-a752-d49b305fd470-kube-api-access-8bxxw\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.885477 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.885392 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c2bdce0-1beb-425b-a752-d49b305fd470-tmp\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.986398 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.986352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxxw\" (UniqueName: \"kubernetes.io/projected/8c2bdce0-1beb-425b-a752-d49b305fd470-kube-api-access-8bxxw\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.986606 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.986472 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c2bdce0-1beb-425b-a752-d49b305fd470-tmp\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.986606 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.986502 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2bdce0-1beb-425b-a752-d49b305fd470-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.986728 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:03:35.986612 2550 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 06 14:03:35.986728 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:03:35.986681 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c2bdce0-1beb-425b-a752-d49b305fd470-tls-certs podName:8c2bdce0-1beb-425b-a752-d49b305fd470 nodeName:}" failed. No retries permitted until 2026-09-06 14:03:36.486659252 +0000 UTC m=+365.750103970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8c2bdce0-1beb-425b-a752-d49b305fd470-tls-certs") pod "kube-auth-proxy-86996bf9f5-5l4pj" (UID: "8c2bdce0-1beb-425b-a752-d49b305fd470") : secret "kube-auth-proxy-tls" not found Sep 06 14:03:35.988912 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.988859 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c2bdce0-1beb-425b-a752-d49b305fd470-tmp\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:35.998261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:35.998231 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxxw\" (UniqueName: \"kubernetes.io/projected/8c2bdce0-1beb-425b-a752-d49b305fd470-kube-api-access-8bxxw\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:36.491489 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.491444 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2bdce0-1beb-425b-a752-d49b305fd470-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:36.493990 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.493967 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2bdce0-1beb-425b-a752-d49b305fd470-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-5l4pj\" (UID: \"8c2bdce0-1beb-425b-a752-d49b305fd470\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:36.746726 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.746638 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" Sep 06 14:03:36.950308 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.950269 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp"] Sep 06 14:03:36.954111 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.954088 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.956843 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.956815 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 06 14:03:36.956843 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.956835 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 06 14:03:36.957068 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.956815 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 06 14:03:36.957068 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.956936 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6nxqb\"" Sep 06 14:03:36.957068 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.956966 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 06 14:03:36.966037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.966009 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp"] Sep 06 14:03:36.996532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996493 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5853bdd-7887-4785-a058-3e1e3c61d61f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.996532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996530 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.996715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996617 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.996715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996669 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.996715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996685 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.996715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996713 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:36.996883 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:36.996780 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqc5d\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-kube-api-access-gqc5d\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.019813 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.019786 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj"] Sep 06 14:03:37.021907 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:37.021866 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2bdce0_1beb_425b_a752_d49b305fd470.slice/crio-762bcb8555535272ec0f31071b8780cb431cf4d7481a7e6bd3d8e0586b447840 WatchSource:0}: Error finding container 762bcb8555535272ec0f31071b8780cb431cf4d7481a7e6bd3d8e0586b447840: Status 404 returned error can't find the container with id 762bcb8555535272ec0f31071b8780cb431cf4d7481a7e6bd3d8e0586b447840 Sep 06 14:03:37.097327 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097290 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.097327 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.097618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097355 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.097618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097378 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.097618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097406 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqc5d\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-kube-api-access-gqc5d\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.097618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097432 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5853bdd-7887-4785-a058-3e1e3c61d61f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.097618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.097451 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.098125 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.098096 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.099831 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.099808 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5853bdd-7887-4785-a058-3e1e3c61d61f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.099959 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.099836 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.100192 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.100177 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.100235 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.100193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.105101 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.105077 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.105411 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.105389 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqc5d\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-kube-api-access-gqc5d\") pod \"istiod-openshift-gateway-7b7995d5cd-7w9zp\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.263672 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.263581 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:37.539414 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.539313 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp"] Sep 06 14:03:37.541648 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:37.541620 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5853bdd_7887_4785_a058_3e1e3c61d61f.slice/crio-6955713ebdf0a9ea24673cf695baf9cbe1fec68b9aa334726caf6b0ca973cde9 WatchSource:0}: Error finding container 6955713ebdf0a9ea24673cf695baf9cbe1fec68b9aa334726caf6b0ca973cde9: Status 404 returned error can't find the container with id 6955713ebdf0a9ea24673cf695baf9cbe1fec68b9aa334726caf6b0ca973cde9 Sep 06 14:03:37.938623 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.938548 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" event={"ID":"c5853bdd-7887-4785-a058-3e1e3c61d61f","Type":"ContainerStarted","Data":"6955713ebdf0a9ea24673cf695baf9cbe1fec68b9aa334726caf6b0ca973cde9"} Sep 06 14:03:37.940128 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:37.940097 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" event={"ID":"8c2bdce0-1beb-425b-a752-d49b305fd470","Type":"ContainerStarted","Data":"762bcb8555535272ec0f31071b8780cb431cf4d7481a7e6bd3d8e0586b447840"} Sep 06 14:03:39.954531 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:39.954480 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" event={"ID":"8c2bdce0-1beb-425b-a752-d49b305fd470","Type":"ContainerStarted","Data":"2c6b5f7ffb472cdbd87939fca8bd1bd6fddfc36b602d1f773dfbad2e9708e2f0"} Sep 06 14:03:39.971215 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:39.971156 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-5l4pj" podStartSLOduration=2.270338927 podStartE2EDuration="4.97113312s" totalImagesPullingTime="2.700794193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:39.96824152 +0000 UTC m=+369.231686245" watchObservedRunningTime="2026-09-06 14:03:39.97113312 +0000 UTC m=+369.234577844" Sep 06 14:03:41.545866 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:41.545820 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 06 14:03:41.546306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:41.545926 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 06 14:03:41.966581 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:41.966549 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" event={"ID":"c5853bdd-7887-4785-a058-3e1e3c61d61f","Type":"ContainerStarted","Data":"d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a"} Sep 06 14:03:41.966767 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:41.966622 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:41.984875 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:41.984819 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" podStartSLOduration=1.9830317339999999 podStartE2EDuration="5.984803398s" totalImagesPullingTime="4.001771664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:41.984318635 +0000 UTC m=+371.247763361" watchObservedRunningTime="2026-09-06 14:03:41.984803398 +0000 UTC m=+371.248248122" Sep 06 14:03:42.972981 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:42.972947 2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-7w9zp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 14:03:42.973352 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:42.973022 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" podUID="c5853bdd-7887-4785-a058-3e1e3c61d61f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 14:03:45.971794 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:45.971764 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:56.122086 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.122036 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr"] Sep 06 14:03:56.130712 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.130682 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.133102 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.133077 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 06 14:03:56.133256 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.133121 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-498dm\"" Sep 06 14:03:56.137973 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.137939 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr"] Sep 06 14:03:56.140764 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.140730 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8"] Sep 06 14:03:56.145098 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.145072 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.147587 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.147560 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 06 14:03:56.147754 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.147572 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzzv5\"" Sep 06 14:03:56.155832 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.155800 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8"] Sep 06 14:03:56.271065 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.271021 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ffe7fd6-4919-4657-b2e2-7aa13212f42e-cert\") pod \"kserve-controller-manager-d6f4b65b8-8cbqr\" (UID: \"2ffe7fd6-4919-4657-b2e2-7aa13212f42e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.271065 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.271068 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rll4l\" (UniqueName: \"kubernetes.io/projected/2ffe7fd6-4919-4657-b2e2-7aa13212f42e-kube-api-access-rll4l\") pod \"kserve-controller-manager-d6f4b65b8-8cbqr\" (UID: \"2ffe7fd6-4919-4657-b2e2-7aa13212f42e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.271314 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.271121 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsg6\" (UniqueName: \"kubernetes.io/projected/3040babf-f012-4c50-9998-274aebf42331-kube-api-access-6hsg6\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.271314 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.271177 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3040babf-f012-4c50-9998-274aebf42331-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.372066 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.372031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ffe7fd6-4919-4657-b2e2-7aa13212f42e-cert\") pod \"kserve-controller-manager-d6f4b65b8-8cbqr\" (UID: \"2ffe7fd6-4919-4657-b2e2-7aa13212f42e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.372066 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.372065 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rll4l\" (UniqueName: \"kubernetes.io/projected/2ffe7fd6-4919-4657-b2e2-7aa13212f42e-kube-api-access-rll4l\") pod \"kserve-controller-manager-d6f4b65b8-8cbqr\" (UID: \"2ffe7fd6-4919-4657-b2e2-7aa13212f42e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.372353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.372103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsg6\" (UniqueName: \"kubernetes.io/projected/3040babf-f012-4c50-9998-274aebf42331-kube-api-access-6hsg6\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.372353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.372127 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3040babf-f012-4c50-9998-274aebf42331-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.372353 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:03:56.372230 2550 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 06 14:03:56.372353 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:03:56.372295 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3040babf-f012-4c50-9998-274aebf42331-cert podName:3040babf-f012-4c50-9998-274aebf42331 nodeName:}" failed. No retries permitted until 2026-09-06 14:03:56.872275711 +0000 UTC m=+386.135720427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3040babf-f012-4c50-9998-274aebf42331-cert") pod "llmisvc-controller-manager-777c6bd7d5-h8sc8" (UID: "3040babf-f012-4c50-9998-274aebf42331") : secret "llmisvc-webhook-server-cert" not found Sep 06 14:03:56.374580 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.374554 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ffe7fd6-4919-4657-b2e2-7aa13212f42e-cert\") pod \"kserve-controller-manager-d6f4b65b8-8cbqr\" (UID: \"2ffe7fd6-4919-4657-b2e2-7aa13212f42e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.383094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.383060 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rll4l\" (UniqueName: \"kubernetes.io/projected/2ffe7fd6-4919-4657-b2e2-7aa13212f42e-kube-api-access-rll4l\") pod \"kserve-controller-manager-d6f4b65b8-8cbqr\" (UID: \"2ffe7fd6-4919-4657-b2e2-7aa13212f42e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.383450 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.383429 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsg6\" (UniqueName: \"kubernetes.io/projected/3040babf-f012-4c50-9998-274aebf42331-kube-api-access-6hsg6\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.443542 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.443503 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:03:56.718050 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.718022 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr"] Sep 06 14:03:56.720317 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:56.720288 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffe7fd6_4919_4657_b2e2_7aa13212f42e.slice/crio-040982d0cf25c13031ccc3302e9b7943bc4da5ed23f8c5afbe71efa2fcddbd5b WatchSource:0}: Error finding container 040982d0cf25c13031ccc3302e9b7943bc4da5ed23f8c5afbe71efa2fcddbd5b: Status 404 returned error can't find the container with id 040982d0cf25c13031ccc3302e9b7943bc4da5ed23f8c5afbe71efa2fcddbd5b Sep 06 14:03:56.877001 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.876958 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3040babf-f012-4c50-9998-274aebf42331-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:56.879452 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:56.879414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3040babf-f012-4c50-9998-274aebf42331-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h8sc8\" (UID: \"3040babf-f012-4c50-9998-274aebf42331\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:57.041988 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:57.041879 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" event={"ID":"2ffe7fd6-4919-4657-b2e2-7aa13212f42e","Type":"ContainerStarted","Data":"040982d0cf25c13031ccc3302e9b7943bc4da5ed23f8c5afbe71efa2fcddbd5b"} Sep 06 14:03:57.059870 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:57.059827 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:03:57.337616 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:57.337577 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8"] Sep 06 14:03:57.340336 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:03:57.340301 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3040babf_f012_4c50_9998_274aebf42331.slice/crio-f569fce30f0dd03bbe4c0af5dcc0ab1e0388958040b5b5dbf87e1ae564dd1c2a WatchSource:0}: Error finding container f569fce30f0dd03bbe4c0af5dcc0ab1e0388958040b5b5dbf87e1ae564dd1c2a: Status 404 returned error can't find the container with id f569fce30f0dd03bbe4c0af5dcc0ab1e0388958040b5b5dbf87e1ae564dd1c2a Sep 06 14:03:58.049329 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:58.049293 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" event={"ID":"3040babf-f012-4c50-9998-274aebf42331","Type":"ContainerStarted","Data":"f569fce30f0dd03bbe4c0af5dcc0ab1e0388958040b5b5dbf87e1ae564dd1c2a"} Sep 06 14:03:59.117967 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.117527 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp"] Sep 06 14:03:59.117967 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.117816 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" podUID="c5853bdd-7887-4785-a058-3e1e3c61d61f" containerName="discovery" containerID="cri-o://d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a" gracePeriod=30 Sep 06 14:03:59.430753 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.430727 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:03:59.600613 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600578 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqc5d\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-kube-api-access-gqc5d\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.600786 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600663 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5853bdd-7887-4785-a058-3e1e3c61d61f-local-certs\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.600786 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600687 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-kubeconfig\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.600786 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600776 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-ca-configmap\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.600992 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600800 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-dns-cert\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.600992 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600832 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-cacerts\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.600992 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.600853 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-token\") pod \"c5853bdd-7887-4785-a058-3e1e3c61d61f\" (UID: \"c5853bdd-7887-4785-a058-3e1e3c61d61f\") " Sep 06 14:03:59.601269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.601231 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:03:59.603284 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.603251 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5853bdd-7887-4785-a058-3e1e3c61d61f-local-certs" (OuterVolumeSpecName: "local-certs") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:03:59.603436 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.603409 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:03:59.603512 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.603470 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:03:59.603566 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.603514 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:03:59.603566 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.603550 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-kube-api-access-gqc5d" (OuterVolumeSpecName: "kube-api-access-gqc5d") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "kube-api-access-gqc5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:03:59.603683 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.603669 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-cacerts" (OuterVolumeSpecName: "cacerts") pod "c5853bdd-7887-4785-a058-3e1e3c61d61f" (UID: "c5853bdd-7887-4785-a058-3e1e3c61d61f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:03:59.702017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.701977 2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-ca-configmap\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:59.702017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.702014 2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-csr-dns-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:59.702017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.702028 2550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-cacerts\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:59.702267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.702037 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-token\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:59.702267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.702047 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqc5d\" (UniqueName: \"kubernetes.io/projected/c5853bdd-7887-4785-a058-3e1e3c61d61f-kube-api-access-gqc5d\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:59.702267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.702055 2550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5853bdd-7887-4785-a058-3e1e3c61d61f-local-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:03:59.702267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:03:59.702063 2550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5853bdd-7887-4785-a058-3e1e3c61d61f-istio-kubeconfig\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:00.069530 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.069426 2550 generic.go:356] "Generic (PLEG): container finished" podID="c5853bdd-7887-4785-a058-3e1e3c61d61f" containerID="d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a" exitCode=0 Sep 06 14:04:00.069530 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.069486 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" Sep 06 14:04:00.069530 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.069515 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" event={"ID":"c5853bdd-7887-4785-a058-3e1e3c61d61f","Type":"ContainerDied","Data":"d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a"} Sep 06 14:04:00.069824 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.069570 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp" event={"ID":"c5853bdd-7887-4785-a058-3e1e3c61d61f","Type":"ContainerDied","Data":"6955713ebdf0a9ea24673cf695baf9cbe1fec68b9aa334726caf6b0ca973cde9"} Sep 06 14:04:00.069824 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.069593 2550 scope.go:117] "RemoveContainer" containerID="d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a" Sep 06 14:04:00.072906 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.072857 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" event={"ID":"2ffe7fd6-4919-4657-b2e2-7aa13212f42e","Type":"ContainerStarted","Data":"364b2d452716c6f653d21e84c610a3116325cf927fea513e8f0e8516a1c6318e"} Sep 06 14:04:00.073083 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.073056 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:04:00.089121 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.089065 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" podStartSLOduration=1.797501364 podStartE2EDuration="4.089046818s" totalImagesPullingTime="2.291545454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:04:00.088234647 +0000 UTC m=+389.351679368" watchObservedRunningTime="2026-09-06 14:04:00.089046818 +0000 UTC m=+389.352491544" Sep 06 14:04:00.104026 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.103992 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp"] Sep 06 14:04:00.109850 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.109817 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7w9zp"] Sep 06 14:04:00.331295 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.331270 2550 scope.go:117] "RemoveContainer" containerID="d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a" Sep 06 14:04:00.331691 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:04:00.331667 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a\": container with ID starting with d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a not found: ID does not exist" containerID="d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a" Sep 06 14:04:00.331743 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:00.331705 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a"} err="failed to get container status \"d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a\": rpc error: code = NotFound desc = could not find container \"d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a\": container with ID starting with d39b941d1f6f42c70ad75d3186ac6bc7853556aa125a9214c5ddf27d22d3e27a not found: ID does not exist" Sep 06 14:04:01.080006 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:01.079971 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" event={"ID":"3040babf-f012-4c50-9998-274aebf42331","Type":"ContainerStarted","Data":"19712288b124c98ec614593344de38670c9523fc5c2c2bd7ac2fa908686f1d1d"} Sep 06 14:04:01.094376 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:01.094324 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" podStartSLOduration=2.055537585 podStartE2EDuration="5.094305088s" totalImagesPullingTime="3.038767503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:04:01.092945076 +0000 UTC m=+390.356389797" watchObservedRunningTime="2026-09-06 14:04:01.094305088 +0000 UTC m=+390.357749811" Sep 06 14:04:01.235693 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:01.235654 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5853bdd-7887-4785-a058-3e1e3c61d61f" path="/var/lib/kubelet/pods/c5853bdd-7887-4785-a058-3e1e3c61d61f/volumes" Sep 06 14:04:02.084908 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:02.084850 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:04:06.054909 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.054855 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-847f87b568-vf7t2"] Sep 06 14:04:06.055915 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.055872 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5853bdd-7887-4785-a058-3e1e3c61d61f" containerName="discovery" Sep 06 14:04:06.056079 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.056065 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5853bdd-7887-4785-a058-3e1e3c61d61f" containerName="discovery" Sep 06 14:04:06.056278 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.056263 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5853bdd-7887-4785-a058-3e1e3c61d61f" containerName="discovery" Sep 06 14:04:06.060278 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.060248 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.066166 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.066131 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847f87b568-vf7t2"] Sep 06 14:04:06.150882 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.150838 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-oauth-config\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.151084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.150920 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-config\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.151084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.151004 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-serving-cert\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.151084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.151040 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-trusted-ca-bundle\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.151084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.151063 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-oauth-serving-cert\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.151237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.151115 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kvz\" (UniqueName: \"kubernetes.io/projected/bbf81eea-6288-4204-83b8-ea3c13b25c9f-kube-api-access-86kvz\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.151237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.151148 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-service-ca\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252382 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-serving-cert\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252382 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252386 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-trusted-ca-bundle\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252411 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-oauth-serving-cert\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252460 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kvz\" (UniqueName: \"kubernetes.io/projected/bbf81eea-6288-4204-83b8-ea3c13b25c9f-kube-api-access-86kvz\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252495 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-service-ca\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252537 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-oauth-config\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.252649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.252578 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-config\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.253374 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.253349 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-config\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.253460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.253397 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-oauth-serving-cert\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.253460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.253401 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-service-ca\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.253460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.253447 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf81eea-6288-4204-83b8-ea3c13b25c9f-trusted-ca-bundle\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.255052 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.255029 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-serving-cert\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.255162 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.255116 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbf81eea-6288-4204-83b8-ea3c13b25c9f-console-oauth-config\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.259628 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.259593 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kvz\" (UniqueName: \"kubernetes.io/projected/bbf81eea-6288-4204-83b8-ea3c13b25c9f-kube-api-access-86kvz\") pod \"console-847f87b568-vf7t2\" (UID: \"bbf81eea-6288-4204-83b8-ea3c13b25c9f\") " pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.373326 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.373228 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:06.646713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:06.646680 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847f87b568-vf7t2"] Sep 06 14:04:06.649653 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:04:06.649624 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf81eea_6288_4204_83b8_ea3c13b25c9f.slice/crio-df861744e68761268ad57c3acd8bb3bcc8d88fc6ca2787372548158769ed3360 WatchSource:0}: Error finding container df861744e68761268ad57c3acd8bb3bcc8d88fc6ca2787372548158769ed3360: Status 404 returned error can't find the container with id df861744e68761268ad57c3acd8bb3bcc8d88fc6ca2787372548158769ed3360 Sep 06 14:04:07.112990 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:07.112946 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847f87b568-vf7t2" event={"ID":"bbf81eea-6288-4204-83b8-ea3c13b25c9f","Type":"ContainerStarted","Data":"d2dcce842e1f979fe98217d9afe40a463db2c8b77bf805490da341a4af94d937"} Sep 06 14:04:07.112990 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:07.112997 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847f87b568-vf7t2" event={"ID":"bbf81eea-6288-4204-83b8-ea3c13b25c9f","Type":"ContainerStarted","Data":"df861744e68761268ad57c3acd8bb3bcc8d88fc6ca2787372548158769ed3360"} Sep 06 14:04:07.128860 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:07.128798 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-847f87b568-vf7t2" podStartSLOduration=1.128778045 podStartE2EDuration="1.128778045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:04:07.12835082 +0000 UTC m=+396.391795545" watchObservedRunningTime="2026-09-06 14:04:07.128778045 +0000 UTC m=+396.392222770" Sep 06 14:04:16.373821 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:16.373774 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:16.374361 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:16.373884 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:16.378917 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:16.378879 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:17.173023 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:17.172993 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-847f87b568-vf7t2" Sep 06 14:04:17.215221 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:17.215182 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7854d57d7b-mv5vh"] Sep 06 14:04:31.085760 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:31.085729 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8cbqr" Sep 06 14:04:33.092664 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:33.092631 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h8sc8" Sep 06 14:04:42.237329 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.237262 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7854d57d7b-mv5vh" podUID="961e1d56-71c4-4833-8db7-bf7ff3e92277" containerName="console" containerID="cri-o://8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11" gracePeriod=15 Sep 06 14:04:42.485812 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.485785 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:04:42.557150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557052 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-config\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557094 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-oauth-config\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557154 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsqr9\" (UniqueName: \"kubernetes.io/projected/961e1d56-71c4-4833-8db7-bf7ff3e92277-kube-api-access-fsqr9\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557202 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-service-ca\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557233 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-serving-cert\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557258 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-oauth-serving-cert\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557291 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-trusted-ca-bundle\") pod \"961e1d56-71c4-4833-8db7-bf7ff3e92277\" (UID: \"961e1d56-71c4-4833-8db7-bf7ff3e92277\") " Sep 06 14:04:42.557604 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557487 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-config" (OuterVolumeSpecName: "console-config") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:04:42.557760 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557731 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:04:42.557824 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557759 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-service-ca" (OuterVolumeSpecName: "service-ca") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:04:42.557919 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.557860 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:04:42.559427 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.559397 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961e1d56-71c4-4833-8db7-bf7ff3e92277-kube-api-access-fsqr9" (OuterVolumeSpecName: "kube-api-access-fsqr9") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "kube-api-access-fsqr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:04:42.559538 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.559434 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:04:42.559538 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.559454 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "961e1d56-71c4-4833-8db7-bf7ff3e92277" (UID: "961e1d56-71c4-4833-8db7-bf7ff3e92277"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:04:42.658182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658140 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqr9\" (UniqueName: \"kubernetes.io/projected/961e1d56-71c4-4833-8db7-bf7ff3e92277-kube-api-access-fsqr9\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:42.658182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658175 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:42.658182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658185 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:42.658424 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658195 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-oauth-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:42.658424 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658203 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-trusted-ca-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:42.658424 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658212 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:42.658424 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:42.658220 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/961e1d56-71c4-4833-8db7-bf7ff3e92277-console-oauth-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:43.294877 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.294843 2550 generic.go:356] "Generic (PLEG): container finished" podID="961e1d56-71c4-4833-8db7-bf7ff3e92277" containerID="8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11" exitCode=0 Sep 06 14:04:43.295311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.294906 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7854d57d7b-mv5vh" Sep 06 14:04:43.295311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.294922 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7854d57d7b-mv5vh" event={"ID":"961e1d56-71c4-4833-8db7-bf7ff3e92277","Type":"ContainerDied","Data":"8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11"} Sep 06 14:04:43.295311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.294971 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7854d57d7b-mv5vh" event={"ID":"961e1d56-71c4-4833-8db7-bf7ff3e92277","Type":"ContainerDied","Data":"62d7a84f879a678d8733ed3da3a211f66e659022bf513c9a7dc8b3c36736494e"} Sep 06 14:04:43.295311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.294991 2550 scope.go:117] "RemoveContainer" containerID="8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11" Sep 06 14:04:43.303743 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.303717 2550 scope.go:117] "RemoveContainer" containerID="8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11" Sep 06 14:04:43.304078 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:04:43.304057 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11\": container with ID starting with 8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11 not found: ID does not exist" containerID="8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11" Sep 06 14:04:43.304153 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.304091 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11"} err="failed to get container status \"8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11\": rpc error: code = NotFound desc = could not find container \"8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11\": container with ID starting with 8949c5979ed360bd9c81eed9291e5046b4bb55d35e10886e47871cb23d49fe11 not found: ID does not exist" Sep 06 14:04:43.309037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.309006 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7854d57d7b-mv5vh"] Sep 06 14:04:43.311247 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:43.311224 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7854d57d7b-mv5vh"] Sep 06 14:04:45.235681 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:45.235645 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="961e1d56-71c4-4833-8db7-bf7ff3e92277" path="/var/lib/kubelet/pods/961e1d56-71c4-4833-8db7-bf7ff3e92277/volumes" Sep 06 14:04:50.468845 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.468812 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt"] Sep 06 14:04:50.469371 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.469165 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="961e1d56-71c4-4833-8db7-bf7ff3e92277" containerName="console" Sep 06 14:04:50.469371 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.469176 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="961e1d56-71c4-4833-8db7-bf7ff3e92277" containerName="console" Sep 06 14:04:50.469371 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.469240 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="961e1d56-71c4-4833-8db7-bf7ff3e92277" containerName="console" Sep 06 14:04:50.473792 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.473770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.476236 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.476175 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 06 14:04:50.476236 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.476180 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 06 14:04:50.477055 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.477034 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m7s6n\"" Sep 06 14:04:50.479759 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.479738 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt"] Sep 06 14:04:50.577112 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.577068 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm"] Sep 06 14:04:50.580562 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.580536 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.588278 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.588242 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm"] Sep 06 14:04:50.628185 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.628145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7cp5\" (UniqueName: \"kubernetes.io/projected/36042feb-9aee-4fc0-aab5-70893d5f76eb-kube-api-access-b7cp5\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.628185 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.628188 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.628401 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.628296 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.669855 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.669813 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6"] Sep 06 14:04:50.673352 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.673331 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.680497 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.680464 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6"] Sep 06 14:04:50.729799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.729706 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.729799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.729756 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.729799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.729776 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlj44\" (UniqueName: \"kubernetes.io/projected/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-kube-api-access-mlj44\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.730057 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.729837 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7cp5\" (UniqueName: \"kubernetes.io/projected/36042feb-9aee-4fc0-aab5-70893d5f76eb-kube-api-access-b7cp5\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.730057 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.729863 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.730057 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.729931 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.730296 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.730277 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.730342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.730304 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.737770 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.737737 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7cp5\" (UniqueName: \"kubernetes.io/projected/36042feb-9aee-4fc0-aab5-70893d5f76eb-kube-api-access-b7cp5\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.770917 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.770869 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x"] Sep 06 14:04:50.774667 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.774647 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:50.781409 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.781376 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x"] Sep 06 14:04:50.784850 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.784821 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:50.830734 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.830701 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.830939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.830806 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj44\" (UniqueName: \"kubernetes.io/projected/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-kube-api-access-mlj44\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.830939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.830852 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7klj\" (UniqueName: \"kubernetes.io/projected/c3f248c1-5138-441d-8e4f-bac24ec1cba4-kube-api-access-z7klj\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.830939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.830875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.831109 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.830969 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.831109 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.831014 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.831203 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.831152 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.831320 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.831295 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.839189 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.839156 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj44\" (UniqueName: \"kubernetes.io/projected/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-kube-api-access-mlj44\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.891173 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.891133 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:50.931653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.931617 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:50.931653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.931660 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mmb\" (UniqueName: \"kubernetes.io/projected/ba38afc2-c052-4923-982b-ae3df3aa1fe1-kube-api-access-w4mmb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:50.931940 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.931768 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7klj\" (UniqueName: \"kubernetes.io/projected/c3f248c1-5138-441d-8e4f-bac24ec1cba4-kube-api-access-z7klj\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.931940 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.931805 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.931940 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.931859 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:50.931940 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.931912 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.932236 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.932216 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.932302 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.932272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.941015 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.940424 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7klj\" (UniqueName: \"kubernetes.io/projected/c3f248c1-5138-441d-8e4f-bac24ec1cba4-kube-api-access-z7klj\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:50.983466 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:50.983382 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:51.033101 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.033061 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.033238 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.033178 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.033238 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.033222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mmb\" (UniqueName: \"kubernetes.io/projected/ba38afc2-c052-4923-982b-ae3df3aa1fe1-kube-api-access-w4mmb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.033691 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.033626 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.033805 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.033753 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.041015 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.040984 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mmb\" (UniqueName: \"kubernetes.io/projected/ba38afc2-c052-4923-982b-ae3df3aa1fe1-kube-api-access-w4mmb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.059113 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.059086 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt"] Sep 06 14:04:51.062388 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:04:51.062333 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36042feb_9aee_4fc0_aab5_70893d5f76eb.slice/crio-c49607aee9c4fa67bfe03a92e8b0ad742e30f2a0173dcc054368262a7ee6f916 WatchSource:0}: Error finding container c49607aee9c4fa67bfe03a92e8b0ad742e30f2a0173dcc054368262a7ee6f916: Status 404 returned error can't find the container with id c49607aee9c4fa67bfe03a92e8b0ad742e30f2a0173dcc054368262a7ee6f916 Sep 06 14:04:51.085550 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.085523 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:51.166038 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.166009 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm"] Sep 06 14:04:51.169611 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:04:51.169561 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4501e2f_00d2_44fb_a1df_c5b25b7171bd.slice/crio-3760afc447c2e3c6ff2757f679aec90761948db38e224c645e08523aca59bcbb WatchSource:0}: Error finding container 3760afc447c2e3c6ff2757f679aec90761948db38e224c645e08523aca59bcbb: Status 404 returned error can't find the container with id 3760afc447c2e3c6ff2757f679aec90761948db38e224c645e08523aca59bcbb Sep 06 14:04:51.256129 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.256101 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6"] Sep 06 14:04:51.263542 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:04:51.263507 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f248c1_5138_441d_8e4f_bac24ec1cba4.slice/crio-fb28cefceed278f5b6dfbf7692b731212eacb6f6bce7214278d26febf09ffcf0 WatchSource:0}: Error finding container fb28cefceed278f5b6dfbf7692b731212eacb6f6bce7214278d26febf09ffcf0: Status 404 returned error can't find the container with id fb28cefceed278f5b6dfbf7692b731212eacb6f6bce7214278d26febf09ffcf0 Sep 06 14:04:51.336104 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.336075 2550 generic.go:356] "Generic (PLEG): container finished" podID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerID="9ebdf4e6201791268595ed2175aec692d641c0d6630b4926bd9a9e5f4b6e9767" exitCode=0 Sep 06 14:04:51.336274 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.336166 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" event={"ID":"36042feb-9aee-4fc0-aab5-70893d5f76eb","Type":"ContainerDied","Data":"9ebdf4e6201791268595ed2175aec692d641c0d6630b4926bd9a9e5f4b6e9767"} Sep 06 14:04:51.336274 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.336215 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" event={"ID":"36042feb-9aee-4fc0-aab5-70893d5f76eb","Type":"ContainerStarted","Data":"c49607aee9c4fa67bfe03a92e8b0ad742e30f2a0173dcc054368262a7ee6f916"} Sep 06 14:04:51.337945 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.337915 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" event={"ID":"c3f248c1-5138-441d-8e4f-bac24ec1cba4","Type":"ContainerStarted","Data":"387bf21be0a93d35cd1759927b02f5c101c6b9b6e318240c95fac4ece39acd9a"} Sep 06 14:04:51.338046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.337973 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" event={"ID":"c3f248c1-5138-441d-8e4f-bac24ec1cba4","Type":"ContainerStarted","Data":"fb28cefceed278f5b6dfbf7692b731212eacb6f6bce7214278d26febf09ffcf0"} Sep 06 14:04:51.347872 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.347830 2550 generic.go:356] "Generic (PLEG): container finished" podID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerID="b3d544ce3e905af524e47728489a946bb216d8fff4d79cd84571bc589179a3af" exitCode=0 Sep 06 14:04:51.348063 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.347955 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" event={"ID":"e4501e2f-00d2-44fb-a1df-c5b25b7171bd","Type":"ContainerDied","Data":"b3d544ce3e905af524e47728489a946bb216d8fff4d79cd84571bc589179a3af"} Sep 06 14:04:51.348063 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.348043 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" event={"ID":"e4501e2f-00d2-44fb-a1df-c5b25b7171bd","Type":"ContainerStarted","Data":"3760afc447c2e3c6ff2757f679aec90761948db38e224c645e08523aca59bcbb"} Sep 06 14:04:51.364810 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:51.364782 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x"] Sep 06 14:04:51.368168 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:04:51.368136 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba38afc2_c052_4923_982b_ae3df3aa1fe1.slice/crio-63a14c7b0450b4ca6d6831de112900d4084718d09367a49bbad4aefa53889747 WatchSource:0}: Error finding container 63a14c7b0450b4ca6d6831de112900d4084718d09367a49bbad4aefa53889747: Status 404 returned error can't find the container with id 63a14c7b0450b4ca6d6831de112900d4084718d09367a49bbad4aefa53889747 Sep 06 14:04:52.355576 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.355540 2550 generic.go:356] "Generic (PLEG): container finished" podID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerID="387bf21be0a93d35cd1759927b02f5c101c6b9b6e318240c95fac4ece39acd9a" exitCode=0 Sep 06 14:04:52.355994 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.355620 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" event={"ID":"c3f248c1-5138-441d-8e4f-bac24ec1cba4","Type":"ContainerDied","Data":"387bf21be0a93d35cd1759927b02f5c101c6b9b6e318240c95fac4ece39acd9a"} Sep 06 14:04:52.357540 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.357516 2550 generic.go:356] "Generic (PLEG): container finished" podID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerID="f7b75003e3707100b899a7d621de75548ff234267a5ef5f770d6d50f242f7956" exitCode=0 Sep 06 14:04:52.357648 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.357576 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" event={"ID":"e4501e2f-00d2-44fb-a1df-c5b25b7171bd","Type":"ContainerDied","Data":"f7b75003e3707100b899a7d621de75548ff234267a5ef5f770d6d50f242f7956"} Sep 06 14:04:52.359244 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.359217 2550 generic.go:356] "Generic (PLEG): container finished" podID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerID="888a47d1ab90c5e1fa484b17d049a34dc406380da8fa449aeb207ab65adba6fc" exitCode=0 Sep 06 14:04:52.359334 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.359277 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" event={"ID":"ba38afc2-c052-4923-982b-ae3df3aa1fe1","Type":"ContainerDied","Data":"888a47d1ab90c5e1fa484b17d049a34dc406380da8fa449aeb207ab65adba6fc"} Sep 06 14:04:52.359334 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:52.359306 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" event={"ID":"ba38afc2-c052-4923-982b-ae3df3aa1fe1","Type":"ContainerStarted","Data":"63a14c7b0450b4ca6d6831de112900d4084718d09367a49bbad4aefa53889747"} Sep 06 14:04:53.366059 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:53.365971 2550 generic.go:356] "Generic (PLEG): container finished" podID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerID="92563fdbcd2f8422b479ee3d3eb5617bda4c5a040cd0fb2205661b1d9d8a9ab9" exitCode=0 Sep 06 14:04:53.366472 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:53.366064 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" event={"ID":"36042feb-9aee-4fc0-aab5-70893d5f76eb","Type":"ContainerDied","Data":"92563fdbcd2f8422b479ee3d3eb5617bda4c5a040cd0fb2205661b1d9d8a9ab9"} Sep 06 14:04:53.368409 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:53.368382 2550 generic.go:356] "Generic (PLEG): container finished" podID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerID="dc699c279b6db16c2fae749d865870ac11a0aec5f28f6b8d889083b64257c38c" exitCode=0 Sep 06 14:04:53.368514 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:53.368458 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" event={"ID":"e4501e2f-00d2-44fb-a1df-c5b25b7171bd","Type":"ContainerDied","Data":"dc699c279b6db16c2fae749d865870ac11a0aec5f28f6b8d889083b64257c38c"} Sep 06 14:04:54.376408 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.376369 2550 generic.go:356] "Generic (PLEG): container finished" podID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerID="efc4e5fa44d3b2cec3c808588c18665ad07234f057e24b1907e2071c104f5a8d" exitCode=0 Sep 06 14:04:54.376850 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.376456 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" event={"ID":"ba38afc2-c052-4923-982b-ae3df3aa1fe1","Type":"ContainerDied","Data":"efc4e5fa44d3b2cec3c808588c18665ad07234f057e24b1907e2071c104f5a8d"} Sep 06 14:04:54.379139 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.379109 2550 generic.go:356] "Generic (PLEG): container finished" podID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerID="04d3fff5f879072b5158ea1207b3b5fecc4a52aa83a78ce56b3abdf30da808c0" exitCode=0 Sep 06 14:04:54.379248 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.379192 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" event={"ID":"36042feb-9aee-4fc0-aab5-70893d5f76eb","Type":"ContainerDied","Data":"04d3fff5f879072b5158ea1207b3b5fecc4a52aa83a78ce56b3abdf30da808c0"} Sep 06 14:04:54.381298 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.381268 2550 generic.go:356] "Generic (PLEG): container finished" podID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerID="c5db6ebd3120d1d8fc91159f6e0306ff9c4c79d155e5970a34234934bc14a408" exitCode=0 Sep 06 14:04:54.381417 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.381324 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" event={"ID":"c3f248c1-5138-441d-8e4f-bac24ec1cba4","Type":"ContainerDied","Data":"c5db6ebd3120d1d8fc91159f6e0306ff9c4c79d155e5970a34234934bc14a408"} Sep 06 14:04:54.541560 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.541530 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:54.665940 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.665827 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-util\") pod \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " Sep 06 14:04:54.665940 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.665887 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlj44\" (UniqueName: \"kubernetes.io/projected/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-kube-api-access-mlj44\") pod \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " Sep 06 14:04:54.666122 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.665988 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-bundle\") pod \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\" (UID: \"e4501e2f-00d2-44fb-a1df-c5b25b7171bd\") " Sep 06 14:04:54.666456 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.666420 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-bundle" (OuterVolumeSpecName: "bundle") pod "e4501e2f-00d2-44fb-a1df-c5b25b7171bd" (UID: "e4501e2f-00d2-44fb-a1df-c5b25b7171bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:54.668122 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.668091 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-kube-api-access-mlj44" (OuterVolumeSpecName: "kube-api-access-mlj44") pod "e4501e2f-00d2-44fb-a1df-c5b25b7171bd" (UID: "e4501e2f-00d2-44fb-a1df-c5b25b7171bd"). InnerVolumeSpecName "kube-api-access-mlj44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:04:54.670658 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.670633 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-util" (OuterVolumeSpecName: "util") pod "e4501e2f-00d2-44fb-a1df-c5b25b7171bd" (UID: "e4501e2f-00d2-44fb-a1df-c5b25b7171bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:54.766887 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.766846 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:54.766887 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.766882 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlj44\" (UniqueName: \"kubernetes.io/projected/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-kube-api-access-mlj44\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:54.766887 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:54.766919 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4501e2f-00d2-44fb-a1df-c5b25b7171bd-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:55.388483 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.388445 2550 generic.go:356] "Generic (PLEG): container finished" podID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerID="884c6f9b404ddc3848ec4e5e93841882099df9dcd75496f91a0d6c703fa24741" exitCode=0 Sep 06 14:04:55.388872 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.388524 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" event={"ID":"c3f248c1-5138-441d-8e4f-bac24ec1cba4","Type":"ContainerDied","Data":"884c6f9b404ddc3848ec4e5e93841882099df9dcd75496f91a0d6c703fa24741"} Sep 06 14:04:55.390530 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.390512 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" Sep 06 14:04:55.390645 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.390507 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbggprm" event={"ID":"e4501e2f-00d2-44fb-a1df-c5b25b7171bd","Type":"ContainerDied","Data":"3760afc447c2e3c6ff2757f679aec90761948db38e224c645e08523aca59bcbb"} Sep 06 14:04:55.390645 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.390632 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3760afc447c2e3c6ff2757f679aec90761948db38e224c645e08523aca59bcbb" Sep 06 14:04:55.393052 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.393024 2550 generic.go:356] "Generic (PLEG): container finished" podID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerID="a58abe13dd486eea9fe12abc0e8c3a0da919cd521223202bef903786adea3607" exitCode=0 Sep 06 14:04:55.393169 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.393050 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" event={"ID":"ba38afc2-c052-4923-982b-ae3df3aa1fe1","Type":"ContainerDied","Data":"a58abe13dd486eea9fe12abc0e8c3a0da919cd521223202bef903786adea3607"} Sep 06 14:04:55.521927 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.521880 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:55.675096 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.675055 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7cp5\" (UniqueName: \"kubernetes.io/projected/36042feb-9aee-4fc0-aab5-70893d5f76eb-kube-api-access-b7cp5\") pod \"36042feb-9aee-4fc0-aab5-70893d5f76eb\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " Sep 06 14:04:55.675301 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.675118 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-bundle\") pod \"36042feb-9aee-4fc0-aab5-70893d5f76eb\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " Sep 06 14:04:55.675301 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.675149 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-util\") pod \"36042feb-9aee-4fc0-aab5-70893d5f76eb\" (UID: \"36042feb-9aee-4fc0-aab5-70893d5f76eb\") " Sep 06 14:04:55.675833 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.675807 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-bundle" (OuterVolumeSpecName: "bundle") pod "36042feb-9aee-4fc0-aab5-70893d5f76eb" (UID: "36042feb-9aee-4fc0-aab5-70893d5f76eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:55.677333 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.677310 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36042feb-9aee-4fc0-aab5-70893d5f76eb-kube-api-access-b7cp5" (OuterVolumeSpecName: "kube-api-access-b7cp5") pod "36042feb-9aee-4fc0-aab5-70893d5f76eb" (UID: "36042feb-9aee-4fc0-aab5-70893d5f76eb"). InnerVolumeSpecName "kube-api-access-b7cp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:04:55.680744 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.680713 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-util" (OuterVolumeSpecName: "util") pod "36042feb-9aee-4fc0-aab5-70893d5f76eb" (UID: "36042feb-9aee-4fc0-aab5-70893d5f76eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:55.776035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.775935 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7cp5\" (UniqueName: \"kubernetes.io/projected/36042feb-9aee-4fc0-aab5-70893d5f76eb-kube-api-access-b7cp5\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:55.776035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.775978 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:55.776035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:55.775991 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36042feb-9aee-4fc0-aab5-70893d5f76eb-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:56.400799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.400771 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" Sep 06 14:04:56.401258 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.400808 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053582wt" event={"ID":"36042feb-9aee-4fc0-aab5-70893d5f76eb","Type":"ContainerDied","Data":"c49607aee9c4fa67bfe03a92e8b0ad742e30f2a0173dcc054368262a7ee6f916"} Sep 06 14:04:56.401258 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.400861 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c49607aee9c4fa67bfe03a92e8b0ad742e30f2a0173dcc054368262a7ee6f916" Sep 06 14:04:56.560054 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.560030 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:56.562332 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.562310 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:56.684132 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684098 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-bundle\") pod \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " Sep 06 14:04:56.684315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684139 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4mmb\" (UniqueName: \"kubernetes.io/projected/ba38afc2-c052-4923-982b-ae3df3aa1fe1-kube-api-access-w4mmb\") pod \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " Sep 06 14:04:56.684315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684166 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-util\") pod \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " Sep 06 14:04:56.684315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684248 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7klj\" (UniqueName: \"kubernetes.io/projected/c3f248c1-5138-441d-8e4f-bac24ec1cba4-kube-api-access-z7klj\") pod \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\" (UID: \"c3f248c1-5138-441d-8e4f-bac24ec1cba4\") " Sep 06 14:04:56.684315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684280 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-bundle\") pod \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " Sep 06 14:04:56.684315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684308 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-util\") pod \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\" (UID: \"ba38afc2-c052-4923-982b-ae3df3aa1fe1\") " Sep 06 14:04:56.684803 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684745 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-bundle" (OuterVolumeSpecName: "bundle") pod "c3f248c1-5138-441d-8e4f-bac24ec1cba4" (UID: "c3f248c1-5138-441d-8e4f-bac24ec1cba4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:56.684971 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.684939 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-bundle" (OuterVolumeSpecName: "bundle") pod "ba38afc2-c052-4923-982b-ae3df3aa1fe1" (UID: "ba38afc2-c052-4923-982b-ae3df3aa1fe1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:56.686932 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.686884 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f248c1-5138-441d-8e4f-bac24ec1cba4-kube-api-access-z7klj" (OuterVolumeSpecName: "kube-api-access-z7klj") pod "c3f248c1-5138-441d-8e4f-bac24ec1cba4" (UID: "c3f248c1-5138-441d-8e4f-bac24ec1cba4"). InnerVolumeSpecName "kube-api-access-z7klj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:04:56.687066 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.686909 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba38afc2-c052-4923-982b-ae3df3aa1fe1-kube-api-access-w4mmb" (OuterVolumeSpecName: "kube-api-access-w4mmb") pod "ba38afc2-c052-4923-982b-ae3df3aa1fe1" (UID: "ba38afc2-c052-4923-982b-ae3df3aa1fe1"). InnerVolumeSpecName "kube-api-access-w4mmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:04:56.690260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.690201 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-util" (OuterVolumeSpecName: "util") pod "ba38afc2-c052-4923-982b-ae3df3aa1fe1" (UID: "ba38afc2-c052-4923-982b-ae3df3aa1fe1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:56.690260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.690251 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-util" (OuterVolumeSpecName: "util") pod "c3f248c1-5138-441d-8e4f-bac24ec1cba4" (UID: "c3f248c1-5138-441d-8e4f-bac24ec1cba4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:04:56.784972 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.784932 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7klj\" (UniqueName: \"kubernetes.io/projected/c3f248c1-5138-441d-8e4f-bac24ec1cba4-kube-api-access-z7klj\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:56.784972 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.784967 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:56.784972 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.784976 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba38afc2-c052-4923-982b-ae3df3aa1fe1-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:56.784972 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.784984 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:56.785230 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.784992 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4mmb\" (UniqueName: \"kubernetes.io/projected/ba38afc2-c052-4923-982b-ae3df3aa1fe1-kube-api-access-w4mmb\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:56.785230 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:56.785000 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f248c1-5138-441d-8e4f-bac24ec1cba4-util\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:04:57.408994 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:57.408969 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" Sep 06 14:04:57.409435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:57.408958 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0kr7r6" event={"ID":"c3f248c1-5138-441d-8e4f-bac24ec1cba4","Type":"ContainerDied","Data":"fb28cefceed278f5b6dfbf7692b731212eacb6f6bce7214278d26febf09ffcf0"} Sep 06 14:04:57.409435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:57.409097 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb28cefceed278f5b6dfbf7692b731212eacb6f6bce7214278d26febf09ffcf0" Sep 06 14:04:57.411040 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:57.411002 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" Sep 06 14:04:57.411040 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:57.411008 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c72x8x" event={"ID":"ba38afc2-c052-4923-982b-ae3df3aa1fe1","Type":"ContainerDied","Data":"63a14c7b0450b4ca6d6831de112900d4084718d09367a49bbad4aefa53889747"} Sep 06 14:04:57.411200 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:04:57.411062 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a14c7b0450b4ca6d6831de112900d4084718d09367a49bbad4aefa53889747" Sep 06 14:05:03.477098 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477064 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j"] Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477380 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477390 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477397 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477403 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477411 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477417 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477423 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477428 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477435 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477440 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477448 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477453 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477460 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477465 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477473 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477478 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477485 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477490 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477495 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477500 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="util" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477513 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477519 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="pull" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477525 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477530 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477581 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="36042feb-9aee-4fc0-aab5-70893d5f76eb" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477588 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba38afc2-c052-4923-982b-ae3df3aa1fe1" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477595 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4501e2f-00d2-44fb-a1df-c5b25b7171bd" containerName="extract" Sep 06 14:05:03.477626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.477601 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f248c1-5138-441d-8e4f-bac24ec1cba4" containerName="extract" Sep 06 14:05:03.481719 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.481698 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.484188 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.484164 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 06 14:05:03.484312 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.484166 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 06 14:05:03.484312 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.484231 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dchdk\"" Sep 06 14:05:03.492359 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.492330 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j"] Sep 06 14:05:03.544396 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.544355 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2213ac51-c085-4499-8bac-cf978e04aef5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.544541 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.544422 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7b7\" (UniqueName: \"kubernetes.io/projected/2213ac51-c085-4499-8bac-cf978e04aef5-kube-api-access-zb7b7\") pod \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.645114 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.645073 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2213ac51-c085-4499-8bac-cf978e04aef5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.645295 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.645137 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7b7\" (UniqueName: \"kubernetes.io/projected/2213ac51-c085-4499-8bac-cf978e04aef5-kube-api-access-zb7b7\") pod \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.645548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.645523 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2213ac51-c085-4499-8bac-cf978e04aef5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.659508 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.659477 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7b7\" (UniqueName: \"kubernetes.io/projected/2213ac51-c085-4499-8bac-cf978e04aef5-kube-api-access-zb7b7\") pod \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:03.793362 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:03.793261 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:04.071796 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:04.071719 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j"] Sep 06 14:05:04.075157 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:04.075122 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2213ac51_c085_4499_8bac_cf978e04aef5.slice/crio-dfbf5ee385357824b227862d9dc11d6f664d720bf51410507ef0582c91985bb3 WatchSource:0}: Error finding container dfbf5ee385357824b227862d9dc11d6f664d720bf51410507ef0582c91985bb3: Status 404 returned error can't find the container with id dfbf5ee385357824b227862d9dc11d6f664d720bf51410507ef0582c91985bb3 Sep 06 14:05:04.447222 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:04.447189 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" event={"ID":"2213ac51-c085-4499-8bac-cf978e04aef5","Type":"ContainerStarted","Data":"dfbf5ee385357824b227862d9dc11d6f664d720bf51410507ef0582c91985bb3"} Sep 06 14:05:05.467026 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.466986 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-c5x97"] Sep 06 14:05:05.470810 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.470780 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:05.473799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.473742 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-87p4n\"" Sep 06 14:05:05.474064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.474047 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 06 14:05:05.482216 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.482130 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-c5x97"] Sep 06 14:05:05.559288 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.559162 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bl7\" (UniqueName: \"kubernetes.io/projected/5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8-kube-api-access-r4bl7\") pod \"dns-operator-controller-manager-5985564776-c5x97\" (UID: \"5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:05.660618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.660575 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bl7\" (UniqueName: \"kubernetes.io/projected/5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8-kube-api-access-r4bl7\") pod \"dns-operator-controller-manager-5985564776-c5x97\" (UID: \"5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:05.672515 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.672472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bl7\" (UniqueName: \"kubernetes.io/projected/5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8-kube-api-access-r4bl7\") pod \"dns-operator-controller-manager-5985564776-c5x97\" (UID: \"5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:05.786116 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:05.786035 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:06.767953 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:06.767926 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-c5x97"] Sep 06 14:05:06.770006 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:06.769968 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5493ae1f_518d_4ce1_8e4e_1aff9a0bd8b8.slice/crio-db28cdcf65ade75077f1e92ffda28bd84db98b7ee0037ad652ce8b78c2a63f32 WatchSource:0}: Error finding container db28cdcf65ade75077f1e92ffda28bd84db98b7ee0037ad652ce8b78c2a63f32: Status 404 returned error can't find the container with id db28cdcf65ade75077f1e92ffda28bd84db98b7ee0037ad652ce8b78c2a63f32 Sep 06 14:05:07.468808 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.468767 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" event={"ID":"5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8","Type":"ContainerStarted","Data":"db28cdcf65ade75077f1e92ffda28bd84db98b7ee0037ad652ce8b78c2a63f32"} Sep 06 14:05:07.671802 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.671765 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk"] Sep 06 14:05:07.680043 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.680006 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:07.682617 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.682578 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-v5kkq\"" Sep 06 14:05:07.684495 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.684465 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk"] Sep 06 14:05:07.776140 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.776038 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dj5\" (UniqueName: \"kubernetes.io/projected/caf0d3c9-fdef-47ee-a427-9fa223e16d4d-kube-api-access-b8dj5\") pod \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" (UID: \"caf0d3c9-fdef-47ee-a427-9fa223e16d4d\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:07.876732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.876695 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dj5\" (UniqueName: \"kubernetes.io/projected/caf0d3c9-fdef-47ee-a427-9fa223e16d4d-kube-api-access-b8dj5\") pod \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" (UID: \"caf0d3c9-fdef-47ee-a427-9fa223e16d4d\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:07.891639 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.891600 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dj5\" (UniqueName: \"kubernetes.io/projected/caf0d3c9-fdef-47ee-a427-9fa223e16d4d-kube-api-access-b8dj5\") pod \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" (UID: \"caf0d3c9-fdef-47ee-a427-9fa223e16d4d\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:07.993755 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:07.993714 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:08.325672 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:08.325639 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk"] Sep 06 14:05:08.327961 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:08.327933 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf0d3c9_fdef_47ee_a427_9fa223e16d4d.slice/crio-bbaebe8e63ae69222377a86bd859e6f9d4585aef4f1a73b0d2b92798d6e5415e WatchSource:0}: Error finding container bbaebe8e63ae69222377a86bd859e6f9d4585aef4f1a73b0d2b92798d6e5415e: Status 404 returned error can't find the container with id bbaebe8e63ae69222377a86bd859e6f9d4585aef4f1a73b0d2b92798d6e5415e Sep 06 14:05:08.477226 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:08.477186 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" event={"ID":"2213ac51-c085-4499-8bac-cf978e04aef5","Type":"ContainerStarted","Data":"6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2"} Sep 06 14:05:08.477412 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:08.477290 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:08.478939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:08.478907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" event={"ID":"caf0d3c9-fdef-47ee-a427-9fa223e16d4d","Type":"ContainerStarted","Data":"bbaebe8e63ae69222377a86bd859e6f9d4585aef4f1a73b0d2b92798d6e5415e"} Sep 06 14:05:08.504695 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:08.504631 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" podStartSLOduration=1.447428398 podStartE2EDuration="5.504606871s" totalImagesPullingTime="4.057178473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:08.501454461 +0000 UTC m=+457.764899187" watchObservedRunningTime="2026-09-06 14:05:08.504606871 +0000 UTC m=+457.768051595" Sep 06 14:05:09.486822 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:09.486783 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" event={"ID":"5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8","Type":"ContainerStarted","Data":"ae046bdbc385e2053fbce9aff95e74d8f20f9aad3d34f7644a40af8b56745ec8"} Sep 06 14:05:09.487240 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:09.487049 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:09.503222 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:09.503104 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" podStartSLOduration=2.266523368 podStartE2EDuration="4.503080737s" totalImagesPullingTime="2.236557369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:09.502573192 +0000 UTC m=+458.766017919" watchObservedRunningTime="2026-09-06 14:05:09.503080737 +0000 UTC m=+458.766525461" Sep 06 14:05:10.359832 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.359796 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-d6ffc"] Sep 06 14:05:10.365441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.365414 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:10.367884 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.367863 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v7mnd\"" Sep 06 14:05:10.373781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.373755 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-d6ffc"] Sep 06 14:05:10.398114 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.398076 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2f2\" (UniqueName: \"kubernetes.io/projected/2f91e93b-7407-430f-93aa-e15b988d83c9-kube-api-access-pq2f2\") pod \"authorino-operator-68d88948d8-d6ffc\" (UID: \"2f91e93b-7407-430f-93aa-e15b988d83c9\") " pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:10.493614 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.493580 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" event={"ID":"caf0d3c9-fdef-47ee-a427-9fa223e16d4d","Type":"ContainerStarted","Data":"cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a"} Sep 06 14:05:10.498812 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.498786 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2f2\" (UniqueName: \"kubernetes.io/projected/2f91e93b-7407-430f-93aa-e15b988d83c9-kube-api-access-pq2f2\") pod \"authorino-operator-68d88948d8-d6ffc\" (UID: \"2f91e93b-7407-430f-93aa-e15b988d83c9\") " pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:10.508637 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.508604 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2f2\" (UniqueName: \"kubernetes.io/projected/2f91e93b-7407-430f-93aa-e15b988d83c9-kube-api-access-pq2f2\") pod \"authorino-operator-68d88948d8-d6ffc\" (UID: \"2f91e93b-7407-430f-93aa-e15b988d83c9\") " pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:10.531700 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.531578 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" podStartSLOduration=1.5817893010000001 podStartE2EDuration="3.531561082s" totalImagesPullingTime="1.949771781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:10.529759196 +0000 UTC m=+459.793203919" watchObservedRunningTime="2026-09-06 14:05:10.531561082 +0000 UTC m=+459.795005805" Sep 06 14:05:10.676784 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.676738 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:10.951444 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:10.951416 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-d6ffc"] Sep 06 14:05:10.953161 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:10.953131 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f91e93b_7407_430f_93aa_e15b988d83c9.slice/crio-81ae456fb5ee79315937e6ad4f479f7feeed12eeac328a2fd659678591521771 WatchSource:0}: Error finding container 81ae456fb5ee79315937e6ad4f479f7feeed12eeac328a2fd659678591521771: Status 404 returned error can't find the container with id 81ae456fb5ee79315937e6ad4f479f7feeed12eeac328a2fd659678591521771 Sep 06 14:05:11.336375 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.336283 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h"] Sep 06 14:05:11.341265 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.341226 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.344028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.344001 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 06 14:05:11.344028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.344025 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 06 14:05:11.344231 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.344120 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6bpnv\"" Sep 06 14:05:11.349164 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.349130 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h"] Sep 06 14:05:11.406758 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.406721 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45nz\" (UniqueName: \"kubernetes.io/projected/7fd340df-cffa-4e0c-acd0-4dff5d500fda-kube-api-access-n45nz\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.406758 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.406765 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fd340df-cffa-4e0c-acd0-4dff5d500fda-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.407060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.406797 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd340df-cffa-4e0c-acd0-4dff5d500fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.500388 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.500357 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" event={"ID":"2f91e93b-7407-430f-93aa-e15b988d83c9","Type":"ContainerStarted","Data":"81ae456fb5ee79315937e6ad4f479f7feeed12eeac328a2fd659678591521771"} Sep 06 14:05:11.500772 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.500451 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:11.507978 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.507945 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fd340df-cffa-4e0c-acd0-4dff5d500fda-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.508156 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.507991 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd340df-cffa-4e0c-acd0-4dff5d500fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.508156 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.508048 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45nz\" (UniqueName: \"kubernetes.io/projected/7fd340df-cffa-4e0c-acd0-4dff5d500fda-kube-api-access-n45nz\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.508228 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:11.508166 2550 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 06 14:05:11.508263 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:11.508231 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd340df-cffa-4e0c-acd0-4dff5d500fda-plugin-serving-cert podName:7fd340df-cffa-4e0c-acd0-4dff5d500fda nodeName:}" failed. No retries permitted until 2026-09-06 14:05:12.008212553 +0000 UTC m=+461.271657263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7fd340df-cffa-4e0c-acd0-4dff5d500fda-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-nnc2h" (UID: "7fd340df-cffa-4e0c-acd0-4dff5d500fda") : secret "plugin-serving-cert" not found Sep 06 14:05:11.508617 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.508599 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fd340df-cffa-4e0c-acd0-4dff5d500fda-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:11.517778 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:11.517756 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45nz\" (UniqueName: \"kubernetes.io/projected/7fd340df-cffa-4e0c-acd0-4dff5d500fda-kube-api-access-n45nz\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:12.012302 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.012247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd340df-cffa-4e0c-acd0-4dff5d500fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:12.015209 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.015176 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd340df-cffa-4e0c-acd0-4dff5d500fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nnc2h\" (UID: \"7fd340df-cffa-4e0c-acd0-4dff5d500fda\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:12.254997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.254953 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" Sep 06 14:05:12.507276 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.507223 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" event={"ID":"2f91e93b-7407-430f-93aa-e15b988d83c9","Type":"ContainerStarted","Data":"f223686765ae655964b8f37cd9149eb85c4acb0bf310c1f338f5dcccd6c1a213"} Sep 06 14:05:12.507746 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.507461 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:12.523983 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.523841 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" podStartSLOduration=1.214063038 podStartE2EDuration="2.523822523s" totalImagesPullingTime="1.309759485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:12.521079513 +0000 UTC m=+461.784524239" watchObservedRunningTime="2026-09-06 14:05:12.523822523 +0000 UTC m=+461.787267246" Sep 06 14:05:12.540783 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:12.540755 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h"] Sep 06 14:05:12.542828 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:12.542790 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fd340df_cffa_4e0c_acd0_4dff5d500fda.slice/crio-8184eb367b4755ef5f4387ca01b68f49f469120dd4d7aeb68e406962247cc1d1 WatchSource:0}: Error finding container 8184eb367b4755ef5f4387ca01b68f49f469120dd4d7aeb68e406962247cc1d1: Status 404 returned error can't find the container with id 8184eb367b4755ef5f4387ca01b68f49f469120dd4d7aeb68e406962247cc1d1 Sep 06 14:05:13.517985 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:13.517945 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" event={"ID":"7fd340df-cffa-4e0c-acd0-4dff5d500fda","Type":"ContainerStarted","Data":"8184eb367b4755ef5f4387ca01b68f49f469120dd4d7aeb68e406962247cc1d1"} Sep 06 14:05:17.549777 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:17.549732 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" event={"ID":"7fd340df-cffa-4e0c-acd0-4dff5d500fda","Type":"ContainerStarted","Data":"f7293b7fe6cd9d6b57acc354527b55e8291a33c3b1163bec1d66bb1639da7685"} Sep 06 14:05:17.568359 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:17.568298 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nnc2h" podStartSLOduration=1.956681562 podStartE2EDuration="6.568280275s" totalImagesPullingTime="4.611598713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:17.56617028 +0000 UTC m=+466.829615004" watchObservedRunningTime="2026-09-06 14:05:17.568280275 +0000 UTC m=+466.831724999" Sep 06 14:05:19.489324 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:19.489291 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:20.495976 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:20.495945 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-c5x97" Sep 06 14:05:21.488528 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.488485 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j"] Sep 06 14:05:21.488828 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.488675 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" containerName="manager" containerID="cri-o://6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2" gracePeriod=2 Sep 06 14:05:21.496912 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.496860 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j"] Sep 06 14:05:21.499045 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.498996 2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2213ac51-c085-4499-8bac-cf978e04aef5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-m2d7j\": pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.507513 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.507481 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk"] Sep 06 14:05:21.507761 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.507736 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" containerName="manager" containerID="cri-o://cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a" gracePeriod=2 Sep 06 14:05:21.509567 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.509521 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.510088 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.510066 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.518866 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.518832 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn"] Sep 06 14:05:21.519437 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.519414 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" containerName="manager" Sep 06 14:05:21.519437 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.519437 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" containerName="manager" Sep 06 14:05:21.519626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.519528 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" containerName="manager" Sep 06 14:05:21.523175 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.523144 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.526366 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.526298 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk"] Sep 06 14:05:21.528981 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.528945 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg"] Sep 06 14:05:21.529351 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.529336 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" containerName="manager" Sep 06 14:05:21.529400 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.529353 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" containerName="manager" Sep 06 14:05:21.529439 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.529415 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" containerName="manager" Sep 06 14:05:21.530624 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.530364 2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"caf0d3c9-fdef-47ee-a427-9fa223e16d4d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-qlfrk\": pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.532289 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.532256 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.532462 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.532442 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:21.534341 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.534310 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.537578 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.537504 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn"] Sep 06 14:05:21.540817 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.540786 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg"] Sep 06 14:05:21.574824 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.574786 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.576711 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.576679 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.606578 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.606533 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8zt\" (UniqueName: \"kubernetes.io/projected/5bcb678a-dc78-45f5-831a-8eafdd9ecaae-kube-api-access-sj8zt\") pod \"limitador-operator-controller-manager-6c549b6d85-2mjfg\" (UID: \"5bcb678a-dc78-45f5-831a-8eafdd9ecaae\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:21.606781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.606605 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vpt\" (UniqueName: \"kubernetes.io/projected/c164f675-5086-4581-9457-e1b6d5fcd4e7-kube-api-access-46vpt\") pod \"kuadrant-operator-controller-manager-58fc56757c-s9gwn\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.606781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.606633 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c164f675-5086-4581-9457-e1b6d5fcd4e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s9gwn\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.707625 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.707581 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vpt\" (UniqueName: \"kubernetes.io/projected/c164f675-5086-4581-9457-e1b6d5fcd4e7-kube-api-access-46vpt\") pod \"kuadrant-operator-controller-manager-58fc56757c-s9gwn\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.707796 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.707644 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c164f675-5086-4581-9457-e1b6d5fcd4e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s9gwn\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.707796 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.707744 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8zt\" (UniqueName: \"kubernetes.io/projected/5bcb678a-dc78-45f5-831a-8eafdd9ecaae-kube-api-access-sj8zt\") pod \"limitador-operator-controller-manager-6c549b6d85-2mjfg\" (UID: \"5bcb678a-dc78-45f5-831a-8eafdd9ecaae\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:21.708516 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.708495 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c164f675-5086-4581-9457-e1b6d5fcd4e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s9gwn\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.717843 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.717804 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8zt\" (UniqueName: \"kubernetes.io/projected/5bcb678a-dc78-45f5-831a-8eafdd9ecaae-kube-api-access-sj8zt\") pod \"limitador-operator-controller-manager-6c549b6d85-2mjfg\" (UID: \"5bcb678a-dc78-45f5-831a-8eafdd9ecaae\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:21.718023 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.717811 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vpt\" (UniqueName: \"kubernetes.io/projected/c164f675-5086-4581-9457-e1b6d5fcd4e7-kube-api-access-46vpt\") pod \"kuadrant-operator-controller-manager-58fc56757c-s9gwn\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.758244 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.758218 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.760291 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.760260 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.760377 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.760298 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.762191 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.762161 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.764067 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.764037 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:21.765801 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:21.765776 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:21.808988 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.808947 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2213ac51-c085-4499-8bac-cf978e04aef5-extensions-socket-volume\") pod \"2213ac51-c085-4499-8bac-cf978e04aef5\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " Sep 06 14:05:21.808988 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.808995 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8dj5\" (UniqueName: \"kubernetes.io/projected/caf0d3c9-fdef-47ee-a427-9fa223e16d4d-kube-api-access-b8dj5\") pod \"caf0d3c9-fdef-47ee-a427-9fa223e16d4d\" (UID: \"caf0d3c9-fdef-47ee-a427-9fa223e16d4d\") " Sep 06 14:05:21.809248 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.809067 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb7b7\" (UniqueName: \"kubernetes.io/projected/2213ac51-c085-4499-8bac-cf978e04aef5-kube-api-access-zb7b7\") pod \"2213ac51-c085-4499-8bac-cf978e04aef5\" (UID: \"2213ac51-c085-4499-8bac-cf978e04aef5\") " Sep 06 14:05:21.809479 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.809453 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2213ac51-c085-4499-8bac-cf978e04aef5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2213ac51-c085-4499-8bac-cf978e04aef5" (UID: "2213ac51-c085-4499-8bac-cf978e04aef5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:05:21.811416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.811385 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2213ac51-c085-4499-8bac-cf978e04aef5-kube-api-access-zb7b7" (OuterVolumeSpecName: "kube-api-access-zb7b7") pod "2213ac51-c085-4499-8bac-cf978e04aef5" (UID: "2213ac51-c085-4499-8bac-cf978e04aef5"). InnerVolumeSpecName "kube-api-access-zb7b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:05:21.811416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.811390 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf0d3c9-fdef-47ee-a427-9fa223e16d4d-kube-api-access-b8dj5" (OuterVolumeSpecName: "kube-api-access-b8dj5") pod "caf0d3c9-fdef-47ee-a427-9fa223e16d4d" (UID: "caf0d3c9-fdef-47ee-a427-9fa223e16d4d"). InnerVolumeSpecName "kube-api-access-b8dj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:05:21.910667 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.910629 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb7b7\" (UniqueName: \"kubernetes.io/projected/2213ac51-c085-4499-8bac-cf978e04aef5-kube-api-access-zb7b7\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:05:21.910667 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.910660 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2213ac51-c085-4499-8bac-cf978e04aef5-extensions-socket-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:05:21.910667 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.910670 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8dj5\" (UniqueName: \"kubernetes.io/projected/caf0d3c9-fdef-47ee-a427-9fa223e16d4d-kube-api-access-b8dj5\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:05:21.935058 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.935018 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:21.944168 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:21.944131 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:22.215064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.215025 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn"] Sep 06 14:05:22.216365 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:22.216338 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc164f675_5086_4581_9457_e1b6d5fcd4e7.slice/crio-4e7bd09b77f12bfd1f881e5de2b5fd98e75f9d070bae5ff5ea7aa3aec9f0d33f WatchSource:0}: Error finding container 4e7bd09b77f12bfd1f881e5de2b5fd98e75f9d070bae5ff5ea7aa3aec9f0d33f: Status 404 returned error can't find the container with id 4e7bd09b77f12bfd1f881e5de2b5fd98e75f9d070bae5ff5ea7aa3aec9f0d33f Sep 06 14:05:22.233317 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.233271 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg"] Sep 06 14:05:22.235857 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:22.235827 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcb678a_dc78_45f5_831a_8eafdd9ecaae.slice/crio-ce746cc7e4219e3cfc52b935329c23da8ee58b5969cdc1641eda7eca5649bf73 WatchSource:0}: Error finding container ce746cc7e4219e3cfc52b935329c23da8ee58b5969cdc1641eda7eca5649bf73: Status 404 returned error can't find the container with id ce746cc7e4219e3cfc52b935329c23da8ee58b5969cdc1641eda7eca5649bf73 Sep 06 14:05:22.581706 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.581665 2550 generic.go:356] "Generic (PLEG): container finished" podID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" containerID="cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a" exitCode=0 Sep 06 14:05:22.582194 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.581734 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:22.582194 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.581778 2550 scope.go:117] "RemoveContainer" containerID="cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a" Sep 06 14:05:22.583842 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.583810 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:22.583995 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.583946 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" event={"ID":"5bcb678a-dc78-45f5-831a-8eafdd9ecaae","Type":"ContainerStarted","Data":"f75258af8be04a94bed7a72b7730df0d6dfdce17696f64290e8e1d1cc1a98502"} Sep 06 14:05:22.584064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.584007 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:22.584064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.584033 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" event={"ID":"5bcb678a-dc78-45f5-831a-8eafdd9ecaae","Type":"ContainerStarted","Data":"ce746cc7e4219e3cfc52b935329c23da8ee58b5969cdc1641eda7eca5649bf73"} Sep 06 14:05:22.584183 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.584159 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp"] Sep 06 14:05:22.585682 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.585652 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:22.586022 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.586001 2550 generic.go:356] "Generic (PLEG): container finished" podID="2213ac51-c085-4499-8bac-cf978e04aef5" containerID="6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2" exitCode=0 Sep 06 14:05:22.586130 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.586113 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:22.588409 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.588386 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:22.588529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.588417 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" event={"ID":"c164f675-5086-4581-9457-e1b6d5fcd4e7","Type":"ContainerStarted","Data":"b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5"} Sep 06 14:05:22.588529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.588440 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" event={"ID":"c164f675-5086-4581-9457-e1b6d5fcd4e7","Type":"ContainerStarted","Data":"4e7bd09b77f12bfd1f881e5de2b5fd98e75f9d070bae5ff5ea7aa3aec9f0d33f"} Sep 06 14:05:22.588529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.588496 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.593604 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.593585 2550 scope.go:117] "RemoveContainer" containerID="cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a" Sep 06 14:05:22.593963 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.593944 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a\": container with ID starting with cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a not found: ID does not exist" containerID="cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a" Sep 06 14:05:22.594062 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.593978 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a"} err="failed to get container status \"cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a\": rpc error: code = NotFound desc = could not find container \"cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a\": container with ID starting with cc87367eb5dcd8bf0592f4b369918c3cbd8cfb9d6e69fd6f4639405466e71c1a not found: ID does not exist" Sep 06 14:05:22.594062 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.594002 2550 scope.go:117] "RemoveContainer" containerID="6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2" Sep 06 14:05:22.598255 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.598229 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp"] Sep 06 14:05:22.606074 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.606046 2550 scope.go:117] "RemoveContainer" containerID="6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2" Sep 06 14:05:22.606705 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.606671 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2\": container with ID starting with 6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2 not found: ID does not exist" containerID="6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2" Sep 06 14:05:22.606809 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.606702 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2"} err="failed to get container status \"6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2\": rpc error: code = NotFound desc = could not find container \"6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2\": container with ID starting with 6438da7327c19e0a9bceac49faef342aecb2c9f1c1616adb1fd5b84cfba383f2 not found: ID does not exist" Sep 06 14:05:22.610630 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.610578 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" podStartSLOduration=1.6105599229999998 podStartE2EDuration="1.610559923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:22.609283504 +0000 UTC m=+471.872728227" watchObservedRunningTime="2026-09-06 14:05:22.610559923 +0000 UTC m=+471.874004647" Sep 06 14:05:22.611369 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.611324 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:22.613062 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.613031 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:22.614773 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.614736 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-m2d7j\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-m2d7j" Sep 06 14:05:22.616572 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:22.616545 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qlfrk\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qlfrk" Sep 06 14:05:22.641957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.641876 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" podStartSLOduration=1.641849868 podStartE2EDuration="1.641849868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:22.639042178 +0000 UTC m=+471.902486904" watchObservedRunningTime="2026-09-06 14:05:22.641849868 +0000 UTC m=+471.905294596" Sep 06 14:05:22.716250 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.716181 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdls9\" (UniqueName: \"kubernetes.io/projected/39609391-2670-4881-86db-a2988228752b-kube-api-access-pdls9\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-nmnnp\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.716452 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.716301 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39609391-2670-4881-86db-a2988228752b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-nmnnp\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.817568 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.817472 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdls9\" (UniqueName: \"kubernetes.io/projected/39609391-2670-4881-86db-a2988228752b-kube-api-access-pdls9\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-nmnnp\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.817568 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.817528 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39609391-2670-4881-86db-a2988228752b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-nmnnp\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.817967 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.817949 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39609391-2670-4881-86db-a2988228752b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-nmnnp\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.825320 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.825285 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdls9\" (UniqueName: \"kubernetes.io/projected/39609391-2670-4881-86db-a2988228752b-kube-api-access-pdls9\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-nmnnp\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:22.905286 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:22.905244 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:23.189792 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.189758 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp"] Sep 06 14:05:23.193390 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:23.193361 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39609391_2670_4881_86db_a2988228752b.slice/crio-b76d4b84ae1843f767f8ab6d554c88e60512f68558cdb084a287c1134b3daeae WatchSource:0}: Error finding container b76d4b84ae1843f767f8ab6d554c88e60512f68558cdb084a287c1134b3daeae: Status 404 returned error can't find the container with id b76d4b84ae1843f767f8ab6d554c88e60512f68558cdb084a287c1134b3daeae Sep 06 14:05:23.237164 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.237132 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2213ac51-c085-4499-8bac-cf978e04aef5" path="/var/lib/kubelet/pods/2213ac51-c085-4499-8bac-cf978e04aef5/volumes" Sep 06 14:05:23.237618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.237579 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf0d3c9-fdef-47ee-a427-9fa223e16d4d" path="/var/lib/kubelet/pods/caf0d3c9-fdef-47ee-a427-9fa223e16d4d/volumes" Sep 06 14:05:23.520441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.520338 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-d6ffc" Sep 06 14:05:23.598723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.598626 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" event={"ID":"39609391-2670-4881-86db-a2988228752b","Type":"ContainerStarted","Data":"6d4c6e2b6ce34a391c857a83e1417e17ddde6fe253bdd8f9ff6e4f1447369533"} Sep 06 14:05:23.599210 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.598982 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:23.599210 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.599020 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" event={"ID":"39609391-2670-4881-86db-a2988228752b","Type":"ContainerStarted","Data":"b76d4b84ae1843f767f8ab6d554c88e60512f68558cdb084a287c1134b3daeae"} Sep 06 14:05:23.615694 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:23.615639 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" podStartSLOduration=1.6156207889999998 podStartE2EDuration="1.615620789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:23.614436835 +0000 UTC m=+472.877881559" watchObservedRunningTime="2026-09-06 14:05:23.615620789 +0000 UTC m=+472.879065544" Sep 06 14:05:33.601308 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:33.601274 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2mjfg" Sep 06 14:05:33.601723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:33.601346 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:34.608262 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:34.608234 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:05:34.660910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:34.660861 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn"] Sep 06 14:05:34.661145 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:34.661114 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" podUID="c164f675-5086-4581-9457-e1b6d5fcd4e7" containerName="manager" containerID="cri-o://b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5" gracePeriod=10 Sep 06 14:05:34.918212 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:34.918184 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:35.035623 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.035586 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c164f675-5086-4581-9457-e1b6d5fcd4e7-extensions-socket-volume\") pod \"c164f675-5086-4581-9457-e1b6d5fcd4e7\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " Sep 06 14:05:35.035799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.035677 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46vpt\" (UniqueName: \"kubernetes.io/projected/c164f675-5086-4581-9457-e1b6d5fcd4e7-kube-api-access-46vpt\") pod \"c164f675-5086-4581-9457-e1b6d5fcd4e7\" (UID: \"c164f675-5086-4581-9457-e1b6d5fcd4e7\") " Sep 06 14:05:35.036035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.036006 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164f675-5086-4581-9457-e1b6d5fcd4e7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c164f675-5086-4581-9457-e1b6d5fcd4e7" (UID: "c164f675-5086-4581-9457-e1b6d5fcd4e7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:05:35.037845 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.037816 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c164f675-5086-4581-9457-e1b6d5fcd4e7-kube-api-access-46vpt" (OuterVolumeSpecName: "kube-api-access-46vpt") pod "c164f675-5086-4581-9457-e1b6d5fcd4e7" (UID: "c164f675-5086-4581-9457-e1b6d5fcd4e7"). InnerVolumeSpecName "kube-api-access-46vpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:05:35.136863 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.136761 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vpt\" (UniqueName: \"kubernetes.io/projected/c164f675-5086-4581-9457-e1b6d5fcd4e7-kube-api-access-46vpt\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:05:35.136863 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.136800 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c164f675-5086-4581-9457-e1b6d5fcd4e7-extensions-socket-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:05:35.669627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.669588 2550 generic.go:356] "Generic (PLEG): container finished" podID="c164f675-5086-4581-9457-e1b6d5fcd4e7" containerID="b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5" exitCode=0 Sep 06 14:05:35.670085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.669654 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" Sep 06 14:05:35.670085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.669668 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" event={"ID":"c164f675-5086-4581-9457-e1b6d5fcd4e7","Type":"ContainerDied","Data":"b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5"} Sep 06 14:05:35.670085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.669730 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn" event={"ID":"c164f675-5086-4581-9457-e1b6d5fcd4e7","Type":"ContainerDied","Data":"4e7bd09b77f12bfd1f881e5de2b5fd98e75f9d070bae5ff5ea7aa3aec9f0d33f"} Sep 06 14:05:35.670085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.669761 2550 scope.go:117] "RemoveContainer" containerID="b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5" Sep 06 14:05:35.678823 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.678802 2550 scope.go:117] "RemoveContainer" containerID="b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5" Sep 06 14:05:35.679279 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:05:35.679258 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5\": container with ID starting with b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5 not found: ID does not exist" containerID="b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5" Sep 06 14:05:35.679373 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.679288 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5"} err="failed to get container status \"b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5\": rpc error: code = NotFound desc = could not find container \"b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5\": container with ID starting with b96d612445c96f622ee31144323aabbd05df6efe866ddd29d6dfe063c79cdaf5 not found: ID does not exist" Sep 06 14:05:35.694698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.694664 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn"] Sep 06 14:05:35.706885 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:35.706852 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s9gwn"] Sep 06 14:05:37.235734 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:37.235701 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c164f675-5086-4581-9457-e1b6d5fcd4e7" path="/var/lib/kubelet/pods/c164f675-5086-4581-9457-e1b6d5fcd4e7/volumes" Sep 06 14:05:55.514349 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.514309 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:05:55.515063 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.514852 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c164f675-5086-4581-9457-e1b6d5fcd4e7" containerName="manager" Sep 06 14:05:55.515063 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.514872 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c164f675-5086-4581-9457-e1b6d5fcd4e7" containerName="manager" Sep 06 14:05:55.515063 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.514986 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c164f675-5086-4581-9457-e1b6d5fcd4e7" containerName="manager" Sep 06 14:05:55.518253 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.518232 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.520442 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.520420 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 06 14:05:55.524543 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.524462 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:05:55.607612 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.607569 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b369a922-9617-4872-bd5c-0f061b5ec88a-config-file\") pod \"limitador-limitador-859c4d96d6-bdd8h\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.607774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.607724 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmt5w\" (UniqueName: \"kubernetes.io/projected/b369a922-9617-4872-bd5c-0f061b5ec88a-kube-api-access-xmt5w\") pod \"limitador-limitador-859c4d96d6-bdd8h\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.617066 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.617027 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:05:55.708449 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.708412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmt5w\" (UniqueName: \"kubernetes.io/projected/b369a922-9617-4872-bd5c-0f061b5ec88a-kube-api-access-xmt5w\") pod \"limitador-limitador-859c4d96d6-bdd8h\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.708660 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.708488 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b369a922-9617-4872-bd5c-0f061b5ec88a-config-file\") pod \"limitador-limitador-859c4d96d6-bdd8h\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.709190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.709171 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b369a922-9617-4872-bd5c-0f061b5ec88a-config-file\") pod \"limitador-limitador-859c4d96d6-bdd8h\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.717057 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.717024 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmt5w\" (UniqueName: \"kubernetes.io/projected/b369a922-9617-4872-bd5c-0f061b5ec88a-kube-api-access-xmt5w\") pod \"limitador-limitador-859c4d96d6-bdd8h\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:55.830295 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:55.830189 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:56.106644 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.106614 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:05:56.109209 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:56.109172 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb369a922_9617_4872_bd5c_0f061b5ec88a.slice/crio-f42c248b76e14275407a90c6bce62b62a27f3d6871bb83c2077687904f53c858 WatchSource:0}: Error finding container f42c248b76e14275407a90c6bce62b62a27f3d6871bb83c2077687904f53c858: Status 404 returned error can't find the container with id f42c248b76e14275407a90c6bce62b62a27f3d6871bb83c2077687904f53c858 Sep 06 14:05:56.398954 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.398866 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-z4lxt"] Sep 06 14:05:56.403613 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.403588 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:05:56.406082 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.406050 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pjvcz\"" Sep 06 14:05:56.409135 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.409101 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z4lxt"] Sep 06 14:05:56.516564 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.516527 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprlv\" (UniqueName: \"kubernetes.io/projected/02e0d4ed-7070-48c2-88b2-b2a6006068f2-kube-api-access-wprlv\") pod \"authorino-b54df94b4-z4lxt\" (UID: \"02e0d4ed-7070-48c2-88b2-b2a6006068f2\") " pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:05:56.618075 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.618029 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprlv\" (UniqueName: \"kubernetes.io/projected/02e0d4ed-7070-48c2-88b2-b2a6006068f2-kube-api-access-wprlv\") pod \"authorino-b54df94b4-z4lxt\" (UID: \"02e0d4ed-7070-48c2-88b2-b2a6006068f2\") " pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:05:56.625336 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.625294 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprlv\" (UniqueName: \"kubernetes.io/projected/02e0d4ed-7070-48c2-88b2-b2a6006068f2-kube-api-access-wprlv\") pod \"authorino-b54df94b4-z4lxt\" (UID: \"02e0d4ed-7070-48c2-88b2-b2a6006068f2\") " pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:05:56.716184 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.716148 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:05:56.789212 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.789177 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" event={"ID":"b369a922-9617-4872-bd5c-0f061b5ec88a","Type":"ContainerStarted","Data":"f42c248b76e14275407a90c6bce62b62a27f3d6871bb83c2077687904f53c858"} Sep 06 14:05:56.988537 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:56.988092 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z4lxt"] Sep 06 14:05:56.992034 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:05:56.991991 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02e0d4ed_7070_48c2_88b2_b2a6006068f2.slice/crio-2b87fd2caf4c6505d566f39750e6d07144d57c928c3b98639d6ae29e2e8a8899 WatchSource:0}: Error finding container 2b87fd2caf4c6505d566f39750e6d07144d57c928c3b98639d6ae29e2e8a8899: Status 404 returned error can't find the container with id 2b87fd2caf4c6505d566f39750e6d07144d57c928c3b98639d6ae29e2e8a8899 Sep 06 14:05:57.800590 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:57.800426 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" event={"ID":"b369a922-9617-4872-bd5c-0f061b5ec88a","Type":"ContainerStarted","Data":"10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e"} Sep 06 14:05:57.801138 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:57.801106 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:05:57.802153 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:57.802111 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z4lxt" event={"ID":"02e0d4ed-7070-48c2-88b2-b2a6006068f2","Type":"ContainerStarted","Data":"2b87fd2caf4c6505d566f39750e6d07144d57c928c3b98639d6ae29e2e8a8899"} Sep 06 14:05:57.817346 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:57.817282 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" podStartSLOduration=1.460865821 podStartE2EDuration="2.817257604s" totalImagesPullingTime="1.356391783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:57.814677189 +0000 UTC m=+507.078121914" watchObservedRunningTime="2026-09-06 14:05:57.817257604 +0000 UTC m=+507.080702331" Sep 06 14:05:59.817035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:59.816988 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z4lxt" event={"ID":"02e0d4ed-7070-48c2-88b2-b2a6006068f2","Type":"ContainerStarted","Data":"313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec"} Sep 06 14:05:59.828961 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:05:59.828885 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-z4lxt" podStartSLOduration=1.91705825 podStartE2EDuration="3.828865369s" totalImagesPullingTime="1.911807119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:05:59.828654923 +0000 UTC m=+509.092099658" watchObservedRunningTime="2026-09-06 14:05:59.828865369 +0000 UTC m=+509.092310094" Sep 06 14:06:00.564102 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:00.564063 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z4lxt"] Sep 06 14:06:01.827585 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:01.827540 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-z4lxt" podUID="02e0d4ed-7070-48c2-88b2-b2a6006068f2" containerName="authorino" containerID="cri-o://313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec" gracePeriod=30 Sep 06 14:06:02.082028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.081950 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:06:02.164454 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.164415 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wprlv\" (UniqueName: \"kubernetes.io/projected/02e0d4ed-7070-48c2-88b2-b2a6006068f2-kube-api-access-wprlv\") pod \"02e0d4ed-7070-48c2-88b2-b2a6006068f2\" (UID: \"02e0d4ed-7070-48c2-88b2-b2a6006068f2\") " Sep 06 14:06:02.166550 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.166520 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e0d4ed-7070-48c2-88b2-b2a6006068f2-kube-api-access-wprlv" (OuterVolumeSpecName: "kube-api-access-wprlv") pod "02e0d4ed-7070-48c2-88b2-b2a6006068f2" (UID: "02e0d4ed-7070-48c2-88b2-b2a6006068f2"). InnerVolumeSpecName "kube-api-access-wprlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:06:02.265825 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.265785 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wprlv\" (UniqueName: \"kubernetes.io/projected/02e0d4ed-7070-48c2-88b2-b2a6006068f2-kube-api-access-wprlv\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:02.834555 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.834517 2550 generic.go:356] "Generic (PLEG): container finished" podID="02e0d4ed-7070-48c2-88b2-b2a6006068f2" containerID="313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec" exitCode=0 Sep 06 14:06:02.835105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.834570 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-z4lxt" Sep 06 14:06:02.835105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.834601 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z4lxt" event={"ID":"02e0d4ed-7070-48c2-88b2-b2a6006068f2","Type":"ContainerDied","Data":"313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec"} Sep 06 14:06:02.835105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.834641 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-z4lxt" event={"ID":"02e0d4ed-7070-48c2-88b2-b2a6006068f2","Type":"ContainerDied","Data":"2b87fd2caf4c6505d566f39750e6d07144d57c928c3b98639d6ae29e2e8a8899"} Sep 06 14:06:02.835105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.834661 2550 scope.go:117] "RemoveContainer" containerID="313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec" Sep 06 14:06:02.844649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.844624 2550 scope.go:117] "RemoveContainer" containerID="313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec" Sep 06 14:06:02.844984 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:06:02.844962 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec\": container with ID starting with 313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec not found: ID does not exist" containerID="313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec" Sep 06 14:06:02.845044 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.844995 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec"} err="failed to get container status \"313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec\": rpc error: code = NotFound desc = could not find container \"313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec\": container with ID starting with 313fe59306dcefe8594aecb22b85b45e289de6d732f3bcb432f445eb59c8f3ec not found: ID does not exist" Sep 06 14:06:02.855113 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.855079 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z4lxt"] Sep 06 14:06:02.858157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:02.858130 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-z4lxt"] Sep 06 14:06:03.235474 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:03.235435 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e0d4ed-7070-48c2-88b2-b2a6006068f2" path="/var/lib/kubelet/pods/02e0d4ed-7070-48c2-88b2-b2a6006068f2/volumes" Sep 06 14:06:09.642276 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:09.642233 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:06:09.642679 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:09.642460 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" podUID="b369a922-9617-4872-bd5c-0f061b5ec88a" containerName="limitador" containerID="cri-o://10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e" gracePeriod=30 Sep 06 14:06:09.643218 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:09.643130 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:06:10.591769 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.591742 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:06:10.742230 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.742137 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmt5w\" (UniqueName: \"kubernetes.io/projected/b369a922-9617-4872-bd5c-0f061b5ec88a-kube-api-access-xmt5w\") pod \"b369a922-9617-4872-bd5c-0f061b5ec88a\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " Sep 06 14:06:10.742230 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.742187 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b369a922-9617-4872-bd5c-0f061b5ec88a-config-file\") pod \"b369a922-9617-4872-bd5c-0f061b5ec88a\" (UID: \"b369a922-9617-4872-bd5c-0f061b5ec88a\") " Sep 06 14:06:10.742638 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.742552 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b369a922-9617-4872-bd5c-0f061b5ec88a-config-file" (OuterVolumeSpecName: "config-file") pod "b369a922-9617-4872-bd5c-0f061b5ec88a" (UID: "b369a922-9617-4872-bd5c-0f061b5ec88a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:06:10.744383 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.744361 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b369a922-9617-4872-bd5c-0f061b5ec88a-kube-api-access-xmt5w" (OuterVolumeSpecName: "kube-api-access-xmt5w") pod "b369a922-9617-4872-bd5c-0f061b5ec88a" (UID: "b369a922-9617-4872-bd5c-0f061b5ec88a"). InnerVolumeSpecName "kube-api-access-xmt5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:06:10.843581 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.843543 2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b369a922-9617-4872-bd5c-0f061b5ec88a-config-file\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:10.843581 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.843574 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmt5w\" (UniqueName: \"kubernetes.io/projected/b369a922-9617-4872-bd5c-0f061b5ec88a-kube-api-access-xmt5w\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:10.881232 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.881194 2550 generic.go:356] "Generic (PLEG): container finished" podID="b369a922-9617-4872-bd5c-0f061b5ec88a" containerID="10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e" exitCode=0 Sep 06 14:06:10.881399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.881242 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" Sep 06 14:06:10.881399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.881244 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" event={"ID":"b369a922-9617-4872-bd5c-0f061b5ec88a","Type":"ContainerDied","Data":"10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e"} Sep 06 14:06:10.881399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.881297 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-bdd8h" event={"ID":"b369a922-9617-4872-bd5c-0f061b5ec88a","Type":"ContainerDied","Data":"f42c248b76e14275407a90c6bce62b62a27f3d6871bb83c2077687904f53c858"} Sep 06 14:06:10.881399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.881325 2550 scope.go:117] "RemoveContainer" containerID="10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e" Sep 06 14:06:10.891337 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.891309 2550 scope.go:117] "RemoveContainer" containerID="10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e" Sep 06 14:06:10.891659 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:06:10.891640 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e\": container with ID starting with 10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e not found: ID does not exist" containerID="10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e" Sep 06 14:06:10.891711 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.891670 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e"} err="failed to get container status \"10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e\": rpc error: code = NotFound desc = could not find container \"10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e\": container with ID starting with 10ae79c27accfac4adfeeac02992c6f6334a461d52d140f69116f50dd213503e not found: ID does not exist" Sep 06 14:06:10.902174 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.902140 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:06:10.905997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:10.905967 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-bdd8h"] Sep 06 14:06:11.235788 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:11.235745 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b369a922-9617-4872-bd5c-0f061b5ec88a" path="/var/lib/kubelet/pods/b369a922-9617-4872-bd5c-0f061b5ec88a/volumes" Sep 06 14:06:12.015066 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015024 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8"] Sep 06 14:06:12.015460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015398 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0d4ed-7070-48c2-88b2-b2a6006068f2" containerName="authorino" Sep 06 14:06:12.015460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015410 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0d4ed-7070-48c2-88b2-b2a6006068f2" containerName="authorino" Sep 06 14:06:12.015460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015419 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b369a922-9617-4872-bd5c-0f061b5ec88a" containerName="limitador" Sep 06 14:06:12.015460 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015425 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b369a922-9617-4872-bd5c-0f061b5ec88a" containerName="limitador" Sep 06 14:06:12.015665 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015496 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e0d4ed-7070-48c2-88b2-b2a6006068f2" containerName="authorino" Sep 06 14:06:12.015665 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.015504 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b369a922-9617-4872-bd5c-0f061b5ec88a" containerName="limitador" Sep 06 14:06:12.019973 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.019948 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.022271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.022229 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 14:06:12.022271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.022252 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 06 14:06:12.023305 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.022954 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 14:06:12.023305 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.023131 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4dxck\"" Sep 06 14:06:12.026335 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.026310 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8"] Sep 06 14:06:12.157450 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.157406 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/71ca09de-9023-4dab-b313-72b2387d2973-data\") pod \"postgres-69fb4594cf-fhcj8\" (UID: \"71ca09de-9023-4dab-b313-72b2387d2973\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.157450 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.157448 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9w7\" (UniqueName: \"kubernetes.io/projected/71ca09de-9023-4dab-b313-72b2387d2973-kube-api-access-bh9w7\") pod \"postgres-69fb4594cf-fhcj8\" (UID: \"71ca09de-9023-4dab-b313-72b2387d2973\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.258707 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.258667 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/71ca09de-9023-4dab-b313-72b2387d2973-data\") pod \"postgres-69fb4594cf-fhcj8\" (UID: \"71ca09de-9023-4dab-b313-72b2387d2973\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.258707 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.258705 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9w7\" (UniqueName: \"kubernetes.io/projected/71ca09de-9023-4dab-b313-72b2387d2973-kube-api-access-bh9w7\") pod \"postgres-69fb4594cf-fhcj8\" (UID: \"71ca09de-9023-4dab-b313-72b2387d2973\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.259102 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.259081 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/71ca09de-9023-4dab-b313-72b2387d2973-data\") pod \"postgres-69fb4594cf-fhcj8\" (UID: \"71ca09de-9023-4dab-b313-72b2387d2973\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.266377 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.266313 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9w7\" (UniqueName: \"kubernetes.io/projected/71ca09de-9023-4dab-b313-72b2387d2973-kube-api-access-bh9w7\") pod \"postgres-69fb4594cf-fhcj8\" (UID: \"71ca09de-9023-4dab-b313-72b2387d2973\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.331816 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.331774 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:12.604937 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.604911 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8"] Sep 06 14:06:12.606537 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:06:12.606503 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ca09de_9023_4dab_b313_72b2387d2973.slice/crio-7631c91dc6754a88c91a83acc87d0efd347688cec22a2224485cff364f80ec32 WatchSource:0}: Error finding container 7631c91dc6754a88c91a83acc87d0efd347688cec22a2224485cff364f80ec32: Status 404 returned error can't find the container with id 7631c91dc6754a88c91a83acc87d0efd347688cec22a2224485cff364f80ec32 Sep 06 14:06:12.894765 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:12.894676 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" event={"ID":"71ca09de-9023-4dab-b313-72b2387d2973","Type":"ContainerStarted","Data":"7631c91dc6754a88c91a83acc87d0efd347688cec22a2224485cff364f80ec32"} Sep 06 14:06:17.929191 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:17.929152 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" event={"ID":"71ca09de-9023-4dab-b313-72b2387d2973","Type":"ContainerStarted","Data":"90a41d002a6de01c5e6b5c8e719b875039ab6b42f4323fa693cc7c158e3e2e58"} Sep 06 14:06:17.929603 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:17.929216 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:17.946157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:17.946101 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" podStartSLOduration=2.047861858 podStartE2EDuration="6.946083043s" totalImagesPullingTime="4.898221185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:06:17.942861196 +0000 UTC m=+527.206305920" watchObservedRunningTime="2026-09-06 14:06:17.946083043 +0000 UTC m=+527.209527767" Sep 06 14:06:23.965152 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:23.965079 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhcj8" Sep 06 14:06:24.482719 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.482675 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qqmlf"] Sep 06 14:06:24.486143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.486118 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:24.488335 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.488316 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pjvcz\"" Sep 06 14:06:24.493727 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.493695 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qqmlf"] Sep 06 14:06:24.661826 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.661791 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qqmlf"] Sep 06 14:06:24.662081 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:06:24.662062 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lwlgg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" podUID="556f54b3-3d03-43d3-8266-78542a78cc40" Sep 06 14:06:24.674697 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.674658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwlgg\" (UniqueName: \"kubernetes.io/projected/556f54b3-3d03-43d3-8266-78542a78cc40-kube-api-access-lwlgg\") pod \"authorino-5dc7ccb5f-qqmlf\" (UID: \"556f54b3-3d03-43d3-8266-78542a78cc40\") " pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:24.687477 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.687433 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-tgqmv"] Sep 06 14:06:24.691035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.691011 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.693373 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.693348 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 06 14:06:24.696988 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.696957 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-tgqmv"] Sep 06 14:06:24.730402 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.730365 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-tgqmv"] Sep 06 14:06:24.730641 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:06:24.730622 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c5tbh tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-c5tbh tls-cert]: context canceled" pod="kuadrant-system/authorino-687bd45585-tgqmv" podUID="d4ed57a7-d124-4688-a3cb-17125c5c12f7" Sep 06 14:06:24.756729 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.756646 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76f76f469b-cl9jk"] Sep 06 14:06:24.760203 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.760177 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:24.765796 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.765770 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76f76f469b-cl9jk"] Sep 06 14:06:24.776094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.776057 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwlgg\" (UniqueName: \"kubernetes.io/projected/556f54b3-3d03-43d3-8266-78542a78cc40-kube-api-access-lwlgg\") pod \"authorino-5dc7ccb5f-qqmlf\" (UID: \"556f54b3-3d03-43d3-8266-78542a78cc40\") " pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:24.783500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.783473 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwlgg\" (UniqueName: \"kubernetes.io/projected/556f54b3-3d03-43d3-8266-78542a78cc40-kube-api-access-lwlgg\") pod \"authorino-5dc7ccb5f-qqmlf\" (UID: \"556f54b3-3d03-43d3-8266-78542a78cc40\") " pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:24.877525 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.877485 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tbh\" (UniqueName: \"kubernetes.io/projected/d4ed57a7-d124-4688-a3cb-17125c5c12f7-kube-api-access-c5tbh\") pod \"authorino-687bd45585-tgqmv\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.877745 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.877543 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4ed57a7-d124-4688-a3cb-17125c5c12f7-tls-cert\") pod \"authorino-687bd45585-tgqmv\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.877745 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.877658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff785199-3619-41d2-af87-99f291d9711e-tls-cert\") pod \"authorino-76f76f469b-cl9jk\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:24.877745 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.877709 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkbx\" (UniqueName: \"kubernetes.io/projected/ff785199-3619-41d2-af87-99f291d9711e-kube-api-access-mvkbx\") pod \"authorino-76f76f469b-cl9jk\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:24.967150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.967116 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:24.967540 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.967116 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.974844 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.974819 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:24.978015 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.977993 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.978368 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.978354 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff785199-3619-41d2-af87-99f291d9711e-tls-cert\") pod \"authorino-76f76f469b-cl9jk\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:24.978418 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.978382 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkbx\" (UniqueName: \"kubernetes.io/projected/ff785199-3619-41d2-af87-99f291d9711e-kube-api-access-mvkbx\") pod \"authorino-76f76f469b-cl9jk\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:24.978418 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.978410 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tbh\" (UniqueName: \"kubernetes.io/projected/d4ed57a7-d124-4688-a3cb-17125c5c12f7-kube-api-access-c5tbh\") pod \"authorino-687bd45585-tgqmv\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.978515 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.978448 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4ed57a7-d124-4688-a3cb-17125c5c12f7-tls-cert\") pod \"authorino-687bd45585-tgqmv\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.981061 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.981029 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4ed57a7-d124-4688-a3cb-17125c5c12f7-tls-cert\") pod \"authorino-687bd45585-tgqmv\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.981197 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.981117 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff785199-3619-41d2-af87-99f291d9711e-tls-cert\") pod \"authorino-76f76f469b-cl9jk\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:24.985848 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.985822 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tbh\" (UniqueName: \"kubernetes.io/projected/d4ed57a7-d124-4688-a3cb-17125c5c12f7-kube-api-access-c5tbh\") pod \"authorino-687bd45585-tgqmv\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:24.986055 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:24.986034 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkbx\" (UniqueName: \"kubernetes.io/projected/ff785199-3619-41d2-af87-99f291d9711e-kube-api-access-mvkbx\") pod \"authorino-76f76f469b-cl9jk\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:25.071089 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.070993 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:06:25.079318 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.079292 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwlgg\" (UniqueName: \"kubernetes.io/projected/556f54b3-3d03-43d3-8266-78542a78cc40-kube-api-access-lwlgg\") pod \"556f54b3-3d03-43d3-8266-78542a78cc40\" (UID: \"556f54b3-3d03-43d3-8266-78542a78cc40\") " Sep 06 14:06:25.079441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.079339 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4ed57a7-d124-4688-a3cb-17125c5c12f7-tls-cert\") pod \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " Sep 06 14:06:25.081523 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.081485 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ed57a7-d124-4688-a3cb-17125c5c12f7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d4ed57a7-d124-4688-a3cb-17125c5c12f7" (UID: "d4ed57a7-d124-4688-a3cb-17125c5c12f7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:06:25.081632 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.081564 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556f54b3-3d03-43d3-8266-78542a78cc40-kube-api-access-lwlgg" (OuterVolumeSpecName: "kube-api-access-lwlgg") pod "556f54b3-3d03-43d3-8266-78542a78cc40" (UID: "556f54b3-3d03-43d3-8266-78542a78cc40"). InnerVolumeSpecName "kube-api-access-lwlgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:06:25.180818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.180776 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5tbh\" (UniqueName: \"kubernetes.io/projected/d4ed57a7-d124-4688-a3cb-17125c5c12f7-kube-api-access-c5tbh\") pod \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\" (UID: \"d4ed57a7-d124-4688-a3cb-17125c5c12f7\") " Sep 06 14:06:25.181046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.180985 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwlgg\" (UniqueName: \"kubernetes.io/projected/556f54b3-3d03-43d3-8266-78542a78cc40-kube-api-access-lwlgg\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:25.181046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.181000 2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4ed57a7-d124-4688-a3cb-17125c5c12f7-tls-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:25.183000 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.182970 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ed57a7-d124-4688-a3cb-17125c5c12f7-kube-api-access-c5tbh" (OuterVolumeSpecName: "kube-api-access-c5tbh") pod "d4ed57a7-d124-4688-a3cb-17125c5c12f7" (UID: "d4ed57a7-d124-4688-a3cb-17125c5c12f7"). InnerVolumeSpecName "kube-api-access-c5tbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:06:25.282079 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.282041 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5tbh\" (UniqueName: \"kubernetes.io/projected/d4ed57a7-d124-4688-a3cb-17125c5c12f7-kube-api-access-c5tbh\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:25.342049 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.342021 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76f76f469b-cl9jk"] Sep 06 14:06:25.344153 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:06:25.344128 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff785199_3619_41d2_af87_99f291d9711e.slice/crio-52eacb7b84f51432f5f81b469ef96e2d03f8019ed5ed0c04b7fed5be7e4d217a WatchSource:0}: Error finding container 52eacb7b84f51432f5f81b469ef96e2d03f8019ed5ed0c04b7fed5be7e4d217a: Status 404 returned error can't find the container with id 52eacb7b84f51432f5f81b469ef96e2d03f8019ed5ed0c04b7fed5be7e4d217a Sep 06 14:06:25.974783 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.974684 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f76f469b-cl9jk" event={"ID":"ff785199-3619-41d2-af87-99f291d9711e","Type":"ContainerStarted","Data":"30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197"} Sep 06 14:06:25.974783 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.974729 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f76f469b-cl9jk" event={"ID":"ff785199-3619-41d2-af87-99f291d9711e","Type":"ContainerStarted","Data":"52eacb7b84f51432f5f81b469ef96e2d03f8019ed5ed0c04b7fed5be7e4d217a"} Sep 06 14:06:25.974783 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.974777 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-tgqmv" Sep 06 14:06:25.975246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.974882 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qqmlf" Sep 06 14:06:25.989757 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:25.989700 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76f76f469b-cl9jk" podStartSLOduration=1.619861754 podStartE2EDuration="1.989681946s" totalImagesPullingTime="369.820192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:06:25.988313518 +0000 UTC m=+535.251758234" watchObservedRunningTime="2026-09-06 14:06:25.989681946 +0000 UTC m=+535.253126733" Sep 06 14:06:26.010689 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:26.010652 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qqmlf"] Sep 06 14:06:26.015384 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:26.015350 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qqmlf"] Sep 06 14:06:26.041689 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:26.041658 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-tgqmv"] Sep 06 14:06:26.045387 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:26.045357 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-tgqmv"] Sep 06 14:06:27.235479 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:27.235448 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="556f54b3-3d03-43d3-8266-78542a78cc40" path="/var/lib/kubelet/pods/556f54b3-3d03-43d3-8266-78542a78cc40/volumes" Sep 06 14:06:27.235968 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:27.235695 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ed57a7-d124-4688-a3cb-17125c5c12f7" path="/var/lib/kubelet/pods/d4ed57a7-d124-4688-a3cb-17125c5c12f7/volumes" Sep 06 14:06:30.557786 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.557746 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fdffcb44b-n7dfz"] Sep 06 14:06:30.741194 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.741160 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fdffcb44b-n7dfz"] Sep 06 14:06:30.741194 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.741202 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c955d44d-xjwbw"] Sep 06 14:06:30.741417 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.741321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.743835 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.743810 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 14:06:30.744018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.743933 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 14:06:30.744945 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.744922 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 14:06:30.745048 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.744958 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d2fsk\"" Sep 06 14:06:30.767027 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.766989 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c955d44d-xjwbw"] Sep 06 14:06:30.767190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.767103 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.824190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.824097 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-webhook-cert\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.824190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.824141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-webhook-cert\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.824384 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.824201 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-metrics-certs\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.824384 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.824243 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqwv\" (UniqueName: \"kubernetes.io/projected/6fc4883d-90de-405e-96e2-0a6e57b4249e-kube-api-access-bgqwv\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.824384 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.824276 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/169e3b99-bc43-4024-8970-936fb971436b-kube-api-access-gskkj\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.824384 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.824342 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-metrics-certs\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.925416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.925370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-webhook-cert\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.925624 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.925430 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-webhook-cert\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.925624 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.925474 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-metrics-certs\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.925624 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.925503 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqwv\" (UniqueName: \"kubernetes.io/projected/6fc4883d-90de-405e-96e2-0a6e57b4249e-kube-api-access-bgqwv\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.925784 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.925695 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/169e3b99-bc43-4024-8970-936fb971436b-kube-api-access-gskkj\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.925784 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.925755 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-metrics-certs\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.928157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.928123 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-webhook-cert\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.928311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.928153 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-metrics-certs\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.928311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.928185 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-webhook-cert\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.928311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.928222 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-metrics-certs\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:30.934279 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.934241 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqwv\" (UniqueName: \"kubernetes.io/projected/6fc4883d-90de-405e-96e2-0a6e57b4249e-kube-api-access-bgqwv\") pod \"maas-controller-6fdffcb44b-n7dfz\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:30.934435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:30.934328 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/169e3b99-bc43-4024-8970-936fb971436b-kube-api-access-gskkj\") pod \"maas-controller-68c955d44d-xjwbw\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:31.052102 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:31.052059 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:31.077784 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:31.077703 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:31.346522 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:31.346497 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fdffcb44b-n7dfz"] Sep 06 14:06:31.348587 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:06:31.348564 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc4883d_90de_405e_96e2_0a6e57b4249e.slice/crio-f9eab1ee70e87b22701343501b228553847c01c07c57465112becdbf8b55709a WatchSource:0}: Error finding container f9eab1ee70e87b22701343501b228553847c01c07c57465112becdbf8b55709a: Status 404 returned error can't find the container with id f9eab1ee70e87b22701343501b228553847c01c07c57465112becdbf8b55709a Sep 06 14:06:31.377982 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:31.377956 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c955d44d-xjwbw"] Sep 06 14:06:31.379628 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:06:31.379597 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169e3b99_bc43_4024_8970_936fb971436b.slice/crio-a4ebd412bc1f263a3098b4a575dc1223cfcf2b27dd1c1f6a732f0298c0da5c8d WatchSource:0}: Error finding container a4ebd412bc1f263a3098b4a575dc1223cfcf2b27dd1c1f6a732f0298c0da5c8d: Status 404 returned error can't find the container with id a4ebd412bc1f263a3098b4a575dc1223cfcf2b27dd1c1f6a732f0298c0da5c8d Sep 06 14:06:32.019065 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:32.018996 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-xjwbw" event={"ID":"169e3b99-bc43-4024-8970-936fb971436b","Type":"ContainerStarted","Data":"a4ebd412bc1f263a3098b4a575dc1223cfcf2b27dd1c1f6a732f0298c0da5c8d"} Sep 06 14:06:32.020663 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:32.020629 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" event={"ID":"6fc4883d-90de-405e-96e2-0a6e57b4249e","Type":"ContainerStarted","Data":"f9eab1ee70e87b22701343501b228553847c01c07c57465112becdbf8b55709a"} Sep 06 14:06:34.000609 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:34.000581 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 14:06:35.040545 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:35.040502 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-xjwbw" event={"ID":"169e3b99-bc43-4024-8970-936fb971436b","Type":"ContainerStarted","Data":"12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d"} Sep 06 14:06:35.041023 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:35.040675 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:35.042185 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:35.042165 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" event={"ID":"6fc4883d-90de-405e-96e2-0a6e57b4249e","Type":"ContainerStarted","Data":"04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c"} Sep 06 14:06:35.042270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:35.042223 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:35.057483 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:35.057426 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c955d44d-xjwbw" podStartSLOduration=2.435995677 podStartE2EDuration="5.057409303s" totalImagesPullingTime="2.621413626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:06:35.055350226 +0000 UTC m=+544.318794951" watchObservedRunningTime="2026-09-06 14:06:35.057409303 +0000 UTC m=+544.320854080" Sep 06 14:06:35.072341 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:35.072287 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" podStartSLOduration=2.424359966 podStartE2EDuration="5.072269696s" totalImagesPullingTime="2.64790973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:06:35.070258128 +0000 UTC m=+544.333702851" watchObservedRunningTime="2026-09-06 14:06:35.072269696 +0000 UTC m=+544.335714421" Sep 06 14:06:46.050261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.050229 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:46.050653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.050300 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:06:46.106993 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.106957 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fdffcb44b-n7dfz"] Sep 06 14:06:46.108187 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.108127 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" podUID="6fc4883d-90de-405e-96e2-0a6e57b4249e" containerName="manager" containerID="cri-o://04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c" gracePeriod=30 Sep 06 14:06:46.356267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.356232 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:46.461210 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.461165 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-webhook-cert\") pod \"6fc4883d-90de-405e-96e2-0a6e57b4249e\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " Sep 06 14:06:46.461210 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.461220 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-metrics-certs\") pod \"6fc4883d-90de-405e-96e2-0a6e57b4249e\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " Sep 06 14:06:46.461430 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.461320 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqwv\" (UniqueName: \"kubernetes.io/projected/6fc4883d-90de-405e-96e2-0a6e57b4249e-kube-api-access-bgqwv\") pod \"6fc4883d-90de-405e-96e2-0a6e57b4249e\" (UID: \"6fc4883d-90de-405e-96e2-0a6e57b4249e\") " Sep 06 14:06:46.463562 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.463528 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6fc4883d-90de-405e-96e2-0a6e57b4249e" (UID: "6fc4883d-90de-405e-96e2-0a6e57b4249e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:06:46.463698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.463585 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc4883d-90de-405e-96e2-0a6e57b4249e-kube-api-access-bgqwv" (OuterVolumeSpecName: "kube-api-access-bgqwv") pod "6fc4883d-90de-405e-96e2-0a6e57b4249e" (UID: "6fc4883d-90de-405e-96e2-0a6e57b4249e"). InnerVolumeSpecName "kube-api-access-bgqwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:06:46.463698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.463619 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6fc4883d-90de-405e-96e2-0a6e57b4249e" (UID: "6fc4883d-90de-405e-96e2-0a6e57b4249e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:06:46.563003 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.562966 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqwv\" (UniqueName: \"kubernetes.io/projected/6fc4883d-90de-405e-96e2-0a6e57b4249e-kube-api-access-bgqwv\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:46.563003 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.563000 2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-webhook-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:46.563003 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:46.563013 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc4883d-90de-405e-96e2-0a6e57b4249e-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:06:47.116053 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.116018 2550 generic.go:356] "Generic (PLEG): container finished" podID="6fc4883d-90de-405e-96e2-0a6e57b4249e" containerID="04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c" exitCode=0 Sep 06 14:06:47.116527 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.116069 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" Sep 06 14:06:47.116527 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.116105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" event={"ID":"6fc4883d-90de-405e-96e2-0a6e57b4249e","Type":"ContainerDied","Data":"04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c"} Sep 06 14:06:47.116527 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.116160 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fdffcb44b-n7dfz" event={"ID":"6fc4883d-90de-405e-96e2-0a6e57b4249e","Type":"ContainerDied","Data":"f9eab1ee70e87b22701343501b228553847c01c07c57465112becdbf8b55709a"} Sep 06 14:06:47.116527 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.116184 2550 scope.go:117] "RemoveContainer" containerID="04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c" Sep 06 14:06:47.126754 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.126736 2550 scope.go:117] "RemoveContainer" containerID="04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c" Sep 06 14:06:47.127111 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:06:47.127091 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c\": container with ID starting with 04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c not found: ID does not exist" containerID="04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c" Sep 06 14:06:47.127159 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.127123 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c"} err="failed to get container status \"04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c\": rpc error: code = NotFound desc = could not find container \"04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c\": container with ID starting with 04b92210d14de5d3863ea69d3ed8de22f8bd65c9dfb252068bd2c87924ee064c not found: ID does not exist" Sep 06 14:06:47.137445 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.137409 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fdffcb44b-n7dfz"] Sep 06 14:06:47.140998 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.140967 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fdffcb44b-n7dfz"] Sep 06 14:06:47.236421 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:06:47.236390 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc4883d-90de-405e-96e2-0a6e57b4249e" path="/var/lib/kubelet/pods/6fc4883d-90de-405e-96e2-0a6e57b4249e/volumes" Sep 06 14:07:08.414330 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.414288 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf"] Sep 06 14:07:08.414866 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.414722 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc4883d-90de-405e-96e2-0a6e57b4249e" containerName="manager" Sep 06 14:07:08.414866 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.414737 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc4883d-90de-405e-96e2-0a6e57b4249e" containerName="manager" Sep 06 14:07:08.414866 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.414802 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc4883d-90de-405e-96e2-0a6e57b4249e" containerName="manager" Sep 06 14:07:08.420177 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.420153 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.423938 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.423886 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 06 14:07:08.423938 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.423885 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 14:07:08.424152 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.423972 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4lj84\"" Sep 06 14:07:08.431583 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.431553 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf"] Sep 06 14:07:08.570856 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.570812 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnqq\" (UniqueName: \"kubernetes.io/projected/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-kube-api-access-5hnqq\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.570856 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.570860 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-openshift-service-ca\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.571113 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.570919 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-metrics-certs\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.571113 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.571000 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-maas-api-tls\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.672479 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.672440 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-maas-api-tls\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.672671 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.672530 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnqq\" (UniqueName: \"kubernetes.io/projected/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-kube-api-access-5hnqq\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.672671 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.672554 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-openshift-service-ca\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.672671 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.672577 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-metrics-certs\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.673242 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.673218 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-openshift-service-ca\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.674945 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.674926 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-metrics-certs\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.674997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.674967 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-maas-api-tls\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.679833 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.679806 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnqq\" (UniqueName: \"kubernetes.io/projected/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-kube-api-access-5hnqq\") pod \"maas-api-7ff94db9d5-jhssf\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.732161 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.732119 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:08.981409 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.981329 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr"] Sep 06 14:07:08.985937 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.985911 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:08.988315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.988281 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 06 14:07:08.988455 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.988318 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pjcd6\"" Sep 06 14:07:08.998051 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:08.998014 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr"] Sep 06 14:07:09.019295 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.018722 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf"] Sep 06 14:07:09.075774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.075736 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d32af-a6e0-4dbf-8800-7685bd8335f3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hlqmr\" (UID: \"013d32af-a6e0-4dbf-8800-7685bd8335f3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.075954 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.075781 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfqk\" (UniqueName: \"kubernetes.io/projected/013d32af-a6e0-4dbf-8800-7685bd8335f3-kube-api-access-wrfqk\") pod \"payload-pre-processing-59fbc47bf8-hlqmr\" (UID: \"013d32af-a6e0-4dbf-8800-7685bd8335f3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.176857 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.176811 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d32af-a6e0-4dbf-8800-7685bd8335f3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hlqmr\" (UID: \"013d32af-a6e0-4dbf-8800-7685bd8335f3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.176857 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.176856 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfqk\" (UniqueName: \"kubernetes.io/projected/013d32af-a6e0-4dbf-8800-7685bd8335f3-kube-api-access-wrfqk\") pod \"payload-pre-processing-59fbc47bf8-hlqmr\" (UID: \"013d32af-a6e0-4dbf-8800-7685bd8335f3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.177626 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.177599 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d32af-a6e0-4dbf-8800-7685bd8335f3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hlqmr\" (UID: \"013d32af-a6e0-4dbf-8800-7685bd8335f3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.184354 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.184318 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfqk\" (UniqueName: \"kubernetes.io/projected/013d32af-a6e0-4dbf-8800-7685bd8335f3-kube-api-access-wrfqk\") pod \"payload-pre-processing-59fbc47bf8-hlqmr\" (UID: \"013d32af-a6e0-4dbf-8800-7685bd8335f3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.249160 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.249048 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" event={"ID":"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12","Type":"ContainerStarted","Data":"6bfd8f2b70c85245635c6bc3d1e12c8fc53849d8fa667624669c12a60b317666"} Sep 06 14:07:09.300684 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.300646 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:09.577870 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:09.577836 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr"] Sep 06 14:07:09.579949 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:07:09.579913 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013d32af_a6e0_4dbf_8800_7685bd8335f3.slice/crio-d82e7e93a5eee8bb52556dd4f4990ee15e7230c8130794fd0bc2f74a2fe5c44d WatchSource:0}: Error finding container d82e7e93a5eee8bb52556dd4f4990ee15e7230c8130794fd0bc2f74a2fe5c44d: Status 404 returned error can't find the container with id d82e7e93a5eee8bb52556dd4f4990ee15e7230c8130794fd0bc2f74a2fe5c44d Sep 06 14:07:10.261641 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:10.261541 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" event={"ID":"013d32af-a6e0-4dbf-8800-7685bd8335f3","Type":"ContainerStarted","Data":"d82e7e93a5eee8bb52556dd4f4990ee15e7230c8130794fd0bc2f74a2fe5c44d"} Sep 06 14:07:11.269271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:11.269234 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" event={"ID":"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12","Type":"ContainerStarted","Data":"d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f"} Sep 06 14:07:11.269776 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:11.269341 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:11.271197 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:11.271170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" event={"ID":"013d32af-a6e0-4dbf-8800-7685bd8335f3","Type":"ContainerStarted","Data":"4f515eb9299f6d54d7f884b80d320809dbda9114cb487e3a4b7aaf68fa8eade0"} Sep 06 14:07:11.271350 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:11.271254 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:11.284462 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:11.284404 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" podStartSLOduration=1.7283567720000002 podStartE2EDuration="3.284382277s" totalImagesPullingTime="1.556025505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:07:11.283949393 +0000 UTC m=+580.547394119" watchObservedRunningTime="2026-09-06 14:07:11.284382277 +0000 UTC m=+580.547827005" Sep 06 14:07:11.299406 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:11.299350 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" podStartSLOduration=1.6955001090000001 podStartE2EDuration="3.299330521s" totalImagesPullingTime="1.603830412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:07:11.297708591 +0000 UTC m=+580.561153342" watchObservedRunningTime="2026-09-06 14:07:11.299330521 +0000 UTC m=+580.562775238" Sep 06 14:07:17.282494 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:17.282457 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:07:22.279085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:22.279047 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hlqmr" Sep 06 14:07:31.209226 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:31.209194 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:07:31.209654 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:31.209194 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:07:32.798025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.797987 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd"] Sep 06 14:07:32.802129 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.802102 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.804664 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.804625 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4dz5\"" Sep 06 14:07:32.804811 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.804715 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 06 14:07:32.804811 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.804749 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 06 14:07:32.805342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.805323 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 06 14:07:32.810196 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.810169 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd"] Sep 06 14:07:32.888567 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.888527 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.888771 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.888586 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.888771 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.888658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.888771 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.888720 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrrq\" (UniqueName: \"kubernetes.io/projected/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-kube-api-access-rfrrq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.888949 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.888800 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.888949 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.888845 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.989558 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.989510 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.989764 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.989574 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.989764 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.989619 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.989764 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.989670 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.989764 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.989701 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.990102 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.990073 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.990213 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.990086 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.990213 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.990158 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.990323 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.990243 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrrq\" (UniqueName: \"kubernetes.io/projected/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-kube-api-access-rfrrq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.992645 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.992616 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.992884 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.992867 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:32.997275 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:32.997245 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrrq\" (UniqueName: \"kubernetes.io/projected/3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8-kube-api-access-rfrrq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd\" (UID: \"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:33.117561 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:33.117462 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:33.398050 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:33.397949 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd"] Sep 06 14:07:33.400993 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:07:33.400959 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f30cbf8_2e79_44d3_bf50_cd46ce4cbad8.slice/crio-f18a89cba16879936f1ebba0d5f588ed689ce67933d32ba51de67988ced9ba24 WatchSource:0}: Error finding container f18a89cba16879936f1ebba0d5f588ed689ce67933d32ba51de67988ced9ba24: Status 404 returned error can't find the container with id f18a89cba16879936f1ebba0d5f588ed689ce67933d32ba51de67988ced9ba24 Sep 06 14:07:34.415105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:34.415071 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" event={"ID":"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8","Type":"ContainerStarted","Data":"f18a89cba16879936f1ebba0d5f588ed689ce67933d32ba51de67988ced9ba24"} Sep 06 14:07:35.424577 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:35.424537 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" event={"ID":"3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8","Type":"ContainerStarted","Data":"3a617daf3aeb6dd62cd6d3fd4213c81745ef7eb7fa7c17185d661a0e7f6823f5"} Sep 06 14:07:35.440835 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:35.440769 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" podStartSLOduration=2.280685607 podStartE2EDuration="3.440750116s" totalImagesPullingTime="1.160064509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:07:35.4389944 +0000 UTC m=+604.702439125" watchObservedRunningTime="2026-09-06 14:07:35.440750116 +0000 UTC m=+604.704194840" Sep 06 14:07:43.118604 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:43.118549 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:43.119081 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:43.118626 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:43.131678 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:43.131646 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:07:43.481555 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:07:43.481519 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd" Sep 06 14:08:24.490332 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.490299 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2"] Sep 06 14:08:24.495292 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.495273 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.498012 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.497987 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 06 14:08:24.503472 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.503442 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2"] Sep 06 14:08:24.557556 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.557524 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.557732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.557563 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.557732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.557637 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnqn\" (UniqueName: \"kubernetes.io/projected/fdf95a9c-ba62-40e1-befb-2867ef33decb-kube-api-access-6rnqn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.557732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.557702 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.557732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.557723 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf95a9c-ba62-40e1-befb-2867ef33decb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.557873 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.557745 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.658605 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.658563 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf95a9c-ba62-40e1-befb-2867ef33decb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.658605 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.658606 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.658989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.658654 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.658989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.658675 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.658989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.658711 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnqn\" (UniqueName: \"kubernetes.io/projected/fdf95a9c-ba62-40e1-befb-2867ef33decb-kube-api-access-6rnqn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.658989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.658777 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.659219 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.659173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.659219 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.659203 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.659330 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.659235 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.661169 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.661138 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf95a9c-ba62-40e1-befb-2867ef33decb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.661353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.661336 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf95a9c-ba62-40e1-befb-2867ef33decb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.666679 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.666647 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnqn\" (UniqueName: \"kubernetes.io/projected/fdf95a9c-ba62-40e1-befb-2867ef33decb-kube-api-access-6rnqn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qbkg2\" (UID: \"fdf95a9c-ba62-40e1-befb-2867ef33decb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:24.808799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:24.808689 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:25.087162 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:25.087130 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2"] Sep 06 14:08:25.089079 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:08:25.089052 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf95a9c_ba62_40e1_befb_2867ef33decb.slice/crio-129bd427013e91239c0bc4ff58f5d926acbd6ec433b5abf23fb5d69d927981a8 WatchSource:0}: Error finding container 129bd427013e91239c0bc4ff58f5d926acbd6ec433b5abf23fb5d69d927981a8: Status 404 returned error can't find the container with id 129bd427013e91239c0bc4ff58f5d926acbd6ec433b5abf23fb5d69d927981a8 Sep 06 14:08:25.090543 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:25.090524 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:08:25.718845 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:25.718809 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" event={"ID":"fdf95a9c-ba62-40e1-befb-2867ef33decb","Type":"ContainerStarted","Data":"be45b464e3dc168888bc0698a70498dad06617fb1294b92d99094a894a0597ae"} Sep 06 14:08:25.719264 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:25.718862 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" event={"ID":"fdf95a9c-ba62-40e1-befb-2867ef33decb","Type":"ContainerStarted","Data":"129bd427013e91239c0bc4ff58f5d926acbd6ec433b5abf23fb5d69d927981a8"} Sep 06 14:08:25.735060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:25.734955 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" podStartSLOduration=1.354521986 podStartE2EDuration="1.734937129s" totalImagesPullingTime="380.415143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:08:25.733254269 +0000 UTC m=+654.996698995" watchObservedRunningTime="2026-09-06 14:08:25.734937129 +0000 UTC m=+654.998381854" Sep 06 14:08:34.809428 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:34.809381 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:34.809855 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:34.809474 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:34.822384 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:34.822346 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:08:35.790655 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:08:35.790623 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qbkg2" Sep 06 14:10:06.429151 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.429107 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-849585cd49-2jzbw"] Sep 06 14:10:06.432977 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.432950 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.441164 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.441117 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-849585cd49-2jzbw"] Sep 06 14:10:06.497102 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.497057 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm54l\" (UniqueName: \"kubernetes.io/projected/b17a3030-3190-4541-93f7-a3e3e7da7a60-kube-api-access-rm54l\") pod \"authorino-849585cd49-2jzbw\" (UID: \"b17a3030-3190-4541-93f7-a3e3e7da7a60\") " pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.497287 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.497116 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b17a3030-3190-4541-93f7-a3e3e7da7a60-tls-cert\") pod \"authorino-849585cd49-2jzbw\" (UID: \"b17a3030-3190-4541-93f7-a3e3e7da7a60\") " pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.598120 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.598078 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b17a3030-3190-4541-93f7-a3e3e7da7a60-tls-cert\") pod \"authorino-849585cd49-2jzbw\" (UID: \"b17a3030-3190-4541-93f7-a3e3e7da7a60\") " pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.598302 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.598193 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm54l\" (UniqueName: \"kubernetes.io/projected/b17a3030-3190-4541-93f7-a3e3e7da7a60-kube-api-access-rm54l\") pod \"authorino-849585cd49-2jzbw\" (UID: \"b17a3030-3190-4541-93f7-a3e3e7da7a60\") " pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.601056 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.601029 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b17a3030-3190-4541-93f7-a3e3e7da7a60-tls-cert\") pod \"authorino-849585cd49-2jzbw\" (UID: \"b17a3030-3190-4541-93f7-a3e3e7da7a60\") " pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.605225 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.605196 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm54l\" (UniqueName: \"kubernetes.io/projected/b17a3030-3190-4541-93f7-a3e3e7da7a60-kube-api-access-rm54l\") pod \"authorino-849585cd49-2jzbw\" (UID: \"b17a3030-3190-4541-93f7-a3e3e7da7a60\") " pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:06.744916 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:06.744783 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-849585cd49-2jzbw" Sep 06 14:10:07.019121 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:07.019088 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-849585cd49-2jzbw"] Sep 06 14:10:07.021117 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:07.021088 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb17a3030_3190_4541_93f7_a3e3e7da7a60.slice/crio-5c0d89a18c4ce85bbb3dbd66679351bbfef5dd9ba27281095b7eb0600e621bab WatchSource:0}: Error finding container 5c0d89a18c4ce85bbb3dbd66679351bbfef5dd9ba27281095b7eb0600e621bab: Status 404 returned error can't find the container with id 5c0d89a18c4ce85bbb3dbd66679351bbfef5dd9ba27281095b7eb0600e621bab Sep 06 14:10:07.321592 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:07.321506 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-849585cd49-2jzbw" event={"ID":"b17a3030-3190-4541-93f7-a3e3e7da7a60","Type":"ContainerStarted","Data":"5c0d89a18c4ce85bbb3dbd66679351bbfef5dd9ba27281095b7eb0600e621bab"} Sep 06 14:10:08.333961 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.333915 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-849585cd49-2jzbw" event={"ID":"b17a3030-3190-4541-93f7-a3e3e7da7a60","Type":"ContainerStarted","Data":"cfdee974cbdb32b32b10d6b1f95648d22c6e8c1bc2249a78cfe665c35037a7b2"} Sep 06 14:10:08.350855 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.350789 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-849585cd49-2jzbw" podStartSLOduration=1.745745879 podStartE2EDuration="2.350766219s" totalImagesPullingTime="605.02034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:08.349940516 +0000 UTC m=+757.613385244" watchObservedRunningTime="2026-09-06 14:10:08.350766219 +0000 UTC m=+757.614210944" Sep 06 14:10:08.378871 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.378707 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76f76f469b-cl9jk"] Sep 06 14:10:08.379421 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.379384 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76f76f469b-cl9jk" podUID="ff785199-3619-41d2-af87-99f291d9711e" containerName="authorino" containerID="cri-o://30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197" gracePeriod=30 Sep 06 14:10:08.648612 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.648310 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:10:08.718253 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.718217 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff785199-3619-41d2-af87-99f291d9711e-tls-cert\") pod \"ff785199-3619-41d2-af87-99f291d9711e\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " Sep 06 14:10:08.718439 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.718318 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvkbx\" (UniqueName: \"kubernetes.io/projected/ff785199-3619-41d2-af87-99f291d9711e-kube-api-access-mvkbx\") pod \"ff785199-3619-41d2-af87-99f291d9711e\" (UID: \"ff785199-3619-41d2-af87-99f291d9711e\") " Sep 06 14:10:08.720395 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.720351 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff785199-3619-41d2-af87-99f291d9711e-kube-api-access-mvkbx" (OuterVolumeSpecName: "kube-api-access-mvkbx") pod "ff785199-3619-41d2-af87-99f291d9711e" (UID: "ff785199-3619-41d2-af87-99f291d9711e"). InnerVolumeSpecName "kube-api-access-mvkbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:10:08.730329 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.730002 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff785199-3619-41d2-af87-99f291d9711e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ff785199-3619-41d2-af87-99f291d9711e" (UID: "ff785199-3619-41d2-af87-99f291d9711e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:10:08.819555 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.819517 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvkbx\" (UniqueName: \"kubernetes.io/projected/ff785199-3619-41d2-af87-99f291d9711e-kube-api-access-mvkbx\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:10:08.819555 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:08.819552 2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff785199-3619-41d2-af87-99f291d9711e-tls-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:10:09.344172 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.344130 2550 generic.go:356] "Generic (PLEG): container finished" podID="ff785199-3619-41d2-af87-99f291d9711e" containerID="30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197" exitCode=0 Sep 06 14:10:09.344653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.344188 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f76f469b-cl9jk" Sep 06 14:10:09.344653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.344221 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f76f469b-cl9jk" event={"ID":"ff785199-3619-41d2-af87-99f291d9711e","Type":"ContainerDied","Data":"30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197"} Sep 06 14:10:09.344653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.344370 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f76f469b-cl9jk" event={"ID":"ff785199-3619-41d2-af87-99f291d9711e","Type":"ContainerDied","Data":"52eacb7b84f51432f5f81b469ef96e2d03f8019ed5ed0c04b7fed5be7e4d217a"} Sep 06 14:10:09.344653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.344396 2550 scope.go:117] "RemoveContainer" containerID="30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197" Sep 06 14:10:09.355353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.355323 2550 scope.go:117] "RemoveContainer" containerID="30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197" Sep 06 14:10:09.355694 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:09.355675 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197\": container with ID starting with 30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197 not found: ID does not exist" containerID="30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197" Sep 06 14:10:09.355745 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.355705 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197"} err="failed to get container status \"30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197\": rpc error: code = NotFound desc = could not find container \"30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197\": container with ID starting with 30ad52ad9c8695595d36586fb1911f32d25bb83a71ce927fce80013eaf017197 not found: ID does not exist" Sep 06 14:10:09.362551 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.362514 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76f76f469b-cl9jk"] Sep 06 14:10:09.367687 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:09.367650 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76f76f469b-cl9jk"] Sep 06 14:10:11.239676 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:11.239645 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff785199-3619-41d2-af87-99f291d9711e" path="/var/lib/kubelet/pods/ff785199-3619-41d2-af87-99f291d9711e/volumes" Sep 06 14:10:27.153942 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.153906 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-140212513809712"] Sep 06 14:10:27.154411 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.154318 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff785199-3619-41d2-af87-99f291d9711e" containerName="authorino" Sep 06 14:10:27.154411 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.154330 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff785199-3619-41d2-af87-99f291d9711e" containerName="authorino" Sep 06 14:10:27.154411 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.154396 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff785199-3619-41d2-af87-99f291d9711e" containerName="authorino" Sep 06 14:10:27.157532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.157505 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:27.160060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.160032 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\"" Sep 06 14:10:27.162812 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.162781 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"] Sep 06 14:10:27.294660 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.294615 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtkv\" (UniqueName: \"kubernetes.io/projected/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65-kube-api-access-7dtkv\") pod \"test-curl-7554-140212513809712\" (UID: \"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65\") " pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:27.396288 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.396240 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtkv\" (UniqueName: \"kubernetes.io/projected/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65-kube-api-access-7dtkv\") pod \"test-curl-7554-140212513809712\" (UID: \"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65\") " pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:27.405152 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.405065 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtkv\" (UniqueName: \"kubernetes.io/projected/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65-kube-api-access-7dtkv\") pod \"test-curl-7554-140212513809712\" (UID: \"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65\") " pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:27.469361 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.469326 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:27.742119 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:27.742078 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"] Sep 06 14:10:27.743264 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:27.743233 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eda2cfe_4dcb_4403_b783_a8bbd1d5cd65.slice/crio-81d16587e8083127bf3a2e44cfaa8099865ae3b7b781535deeadca942f24a7ea WatchSource:0}: Error finding container 81d16587e8083127bf3a2e44cfaa8099865ae3b7b781535deeadca942f24a7ea: Status 404 returned error can't find the container with id 81d16587e8083127bf3a2e44cfaa8099865ae3b7b781535deeadca942f24a7ea Sep 06 14:10:28.462025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:28.461984 2550 generic.go:356] "Generic (PLEG): container finished" podID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" containerID="bb0aecae8fab570cd8c2d1b2f90a1c402cfec2ac5193a1b355bfed3b1e255365" exitCode=0 Sep 06 14:10:28.462449 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:28.462063 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-140212513809712" event={"ID":"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65","Type":"ContainerDied","Data":"bb0aecae8fab570cd8c2d1b2f90a1c402cfec2ac5193a1b355bfed3b1e255365"} Sep 06 14:10:28.462449 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:28.462105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-140212513809712" event={"ID":"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65","Type":"ContainerStarted","Data":"81d16587e8083127bf3a2e44cfaa8099865ae3b7b781535deeadca942f24a7ea"} Sep 06 14:10:29.629540 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.629192 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:29.641459 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.641297 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595"] Sep 06 14:10:29.642511 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.642484 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" containerName="test-curl-7554-140212513809712" Sep 06 14:10:29.642773 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.642745 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" containerName="test-curl-7554-140212513809712" Sep 06 14:10:29.642935 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.642868 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" containerName="test-curl-7554-140212513809712" Sep 06 14:10:29.646632 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.646599 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.649539 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.649501 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1dfd6b-serving-cert\"" Sep 06 14:10:29.658912 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.658134 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595"] Sep 06 14:10:29.667249 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.667217 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7554-140212513809712_3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65/test-curl-7554-140212513809712/0.log" Sep 06 14:10:29.694442 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.694398 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"] Sep 06 14:10:29.697033 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.696994 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"] Sep 06 14:10:29.717303 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.717264 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dtkv\" (UniqueName: \"kubernetes.io/projected/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65-kube-api-access-7dtkv\") pod \"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65\" (UID: \"3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65\") " Sep 06 14:10:29.717563 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.717539 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.717616 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.717577 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-metrics-certs\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.717684 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.717668 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xvl\" (UniqueName: \"kubernetes.io/projected/9045e982-5e27-4dd8-abfa-73fbce19c750-kube-api-access-m8xvl\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.717760 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.717747 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045e982-5e27-4dd8-abfa-73fbce19c750-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.719648 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.719616 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65-kube-api-access-7dtkv" (OuterVolumeSpecName: "kube-api-access-7dtkv") pod "3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" (UID: "3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65"). InnerVolumeSpecName "kube-api-access-7dtkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:10:29.818824 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.818781 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xvl\" (UniqueName: \"kubernetes.io/projected/9045e982-5e27-4dd8-abfa-73fbce19c750-kube-api-access-m8xvl\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.819029 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.818879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045e982-5e27-4dd8-abfa-73fbce19c750-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.819029 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.819007 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.819155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.819031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-metrics-certs\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.819155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.819080 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtkv\" (UniqueName: \"kubernetes.io/projected/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65-kube-api-access-7dtkv\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:10:29.819251 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:29.819181 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-serving-cert: secret "maas-api-e2e-inf-a-1dfd6b-serving-cert" not found Sep 06 14:10:29.819291 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:29.819265 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls podName:9045e982-5e27-4dd8-abfa-73fbce19c750 nodeName:}" failed. No retries permitted until 2026-09-06 14:10:30.319240579 +0000 UTC m=+779.582685285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls") pod "maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" (UID: "9045e982-5e27-4dd8-abfa-73fbce19c750") : secret "maas-api-e2e-inf-a-1dfd6b-serving-cert" not found Sep 06 14:10:29.819644 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.819614 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045e982-5e27-4dd8-abfa-73fbce19c750-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.821805 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.821773 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-metrics-certs\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:29.828875 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:29.828845 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xvl\" (UniqueName: \"kubernetes.io/projected/9045e982-5e27-4dd8-abfa-73fbce19c750-kube-api-access-m8xvl\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:30.323034 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:30.322998 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:30.325481 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:30.325455 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:30.479612 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:30.479584 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:30.479786 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:30.479585 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d16587e8083127bf3a2e44cfaa8099865ae3b7b781535deeadca942f24a7ea" Sep 06 14:10:30.489668 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:30.489633 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7554-140212513809712\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:30.564009 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:30.563969 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:30.843710 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:30.843680 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595"] Sep 06 14:10:30.845949 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:30.845914 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9045e982_5e27_4dd8_abfa_73fbce19c750.slice/crio-3fbcc3b5b2140befd8675eb2c1e980edc23bd6a0e066d45fa00b8c8ec5973fb8 WatchSource:0}: Error finding container 3fbcc3b5b2140befd8675eb2c1e980edc23bd6a0e066d45fa00b8c8ec5973fb8: Status 404 returned error can't find the container with id 3fbcc3b5b2140befd8675eb2c1e980edc23bd6a0e066d45fa00b8c8ec5973fb8 Sep 06 14:10:31.234057 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:31.234009 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7554-140212513809712\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" pod="opendatahub/test-curl-7554-140212513809712" Sep 06 14:10:31.236640 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:31.236609 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65" path="/var/lib/kubelet/pods/3eda2cfe-4dcb-4403-b783-a8bbd1d5cd65/volumes" Sep 06 14:10:31.488382 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:31.488282 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" event={"ID":"9045e982-5e27-4dd8-abfa-73fbce19c750","Type":"ContainerStarted","Data":"7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205"} Sep 06 14:10:31.488382 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:31.488374 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" event={"ID":"9045e982-5e27-4dd8-abfa-73fbce19c750","Type":"ContainerStarted","Data":"3fbcc3b5b2140befd8675eb2c1e980edc23bd6a0e066d45fa00b8c8ec5973fb8"} Sep 06 14:10:31.488581 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:31.488397 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:31.504458 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:31.504392 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" podStartSLOduration=2.100795156 podStartE2EDuration="2.504370796s" totalImagesPullingTime="403.57564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:31.503580728 +0000 UTC m=+780.767025453" watchObservedRunningTime="2026-09-06 14:10:31.504370796 +0000 UTC m=+780.767815520" Sep 06 14:10:37.092542 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.092480 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n"] Sep 06 14:10:37.097432 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.097402 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.099574 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.099544 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-84753aa3-serving-cert\"" Sep 06 14:10:37.109044 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.109016 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n"] Sep 06 14:10:37.189600 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.189559 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-metrics-certs\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.189803 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.189702 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-maas-api-tls\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.189803 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.189761 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtft5\" (UniqueName: \"kubernetes.io/projected/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-kube-api-access-qtft5\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.189803 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.189796 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-openshift-service-ca\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.290949 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.290879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-maas-api-tls\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.291149 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.290961 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtft5\" (UniqueName: \"kubernetes.io/projected/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-kube-api-access-qtft5\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.291149 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.290980 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-openshift-service-ca\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.291149 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.291030 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-metrics-certs\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.291636 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.291608 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-openshift-service-ca\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.293461 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.293436 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-maas-api-tls\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.293570 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.293470 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-metrics-certs\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.298280 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.298251 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtft5\" (UniqueName: \"kubernetes.io/projected/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-kube-api-access-qtft5\") pod \"maas-api-e2e-ait-84753aa3-57d7755c5b-6928n\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.414589 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.414491 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:37.501633 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.501602 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:10:37.700620 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:37.700596 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n"] Sep 06 14:10:37.703121 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:37.703092 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f67d6e0_a5cb_496d_8717_7d2f29c8326a.slice/crio-149706ead74d751010c35732dc2cf91d73e9ccd16a0d7d66b047d7284608f683 WatchSource:0}: Error finding container 149706ead74d751010c35732dc2cf91d73e9ccd16a0d7d66b047d7284608f683: Status 404 returned error can't find the container with id 149706ead74d751010c35732dc2cf91d73e9ccd16a0d7d66b047d7284608f683 Sep 06 14:10:38.536246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:38.536154 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" event={"ID":"1f67d6e0-a5cb-496d-8717-7d2f29c8326a","Type":"ContainerStarted","Data":"d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a"} Sep 06 14:10:38.536246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:38.536212 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" event={"ID":"1f67d6e0-a5cb-496d-8717-7d2f29c8326a","Type":"ContainerStarted","Data":"149706ead74d751010c35732dc2cf91d73e9ccd16a0d7d66b047d7284608f683"} Sep 06 14:10:38.536678 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:38.536318 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:38.553867 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:38.553810 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" podStartSLOduration=1.054255758 podStartE2EDuration="1.553791082s" totalImagesPullingTime="499.535324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:38.552746232 +0000 UTC m=+787.816190952" watchObservedRunningTime="2026-09-06 14:10:38.553791082 +0000 UTC m=+787.817235885" Sep 06 14:10:39.689022 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.688986 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25"] Sep 06 14:10:39.693598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.693570 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.696118 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.696095 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1482ae-serving-cert\"" Sep 06 14:10:39.705710 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.705682 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25"] Sep 06 14:10:39.816555 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.816514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-metrics-certs\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.816555 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.816553 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4vw\" (UniqueName: \"kubernetes.io/projected/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-kube-api-access-4j4vw\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.816781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.816584 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-openshift-service-ca\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.816781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.816624 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-maas-api-tls\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.917377 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.917335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-metrics-certs\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.917377 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.917380 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4vw\" (UniqueName: \"kubernetes.io/projected/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-kube-api-access-4j4vw\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.917649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.917417 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-openshift-service-ca\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.917649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.917588 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-maas-api-tls\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.918138 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.918112 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-openshift-service-ca\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.920117 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.920090 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-metrics-certs\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.920250 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.920229 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-maas-api-tls\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:39.925261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:39.925239 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4vw\" (UniqueName: \"kubernetes.io/projected/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-kube-api-access-4j4vw\") pod \"maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:40.007245 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:40.007152 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:40.291740 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:40.291708 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25"] Sep 06 14:10:40.294072 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:40.294045 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9c575f_a4e6_47c7_a36c_d5d6e3857c7e.slice/crio-7e8d88d857548622149fc3367962b52d1c8021379ca1b2051385a0be564ff84c WatchSource:0}: Error finding container 7e8d88d857548622149fc3367962b52d1c8021379ca1b2051385a0be564ff84c: Status 404 returned error can't find the container with id 7e8d88d857548622149fc3367962b52d1c8021379ca1b2051385a0be564ff84c Sep 06 14:10:40.551506 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:40.551396 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" event={"ID":"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e","Type":"ContainerStarted","Data":"7e8d88d857548622149fc3367962b52d1c8021379ca1b2051385a0be564ff84c"} Sep 06 14:10:41.558427 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:41.558390 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" event={"ID":"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e","Type":"ContainerStarted","Data":"a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235"} Sep 06 14:10:41.558825 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:41.558452 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:41.574595 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:41.574535 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" podStartSLOduration=2.1939851089999998 podStartE2EDuration="2.574516108s" totalImagesPullingTime="380.530999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:41.572349636 +0000 UTC m=+790.835794360" watchObservedRunningTime="2026-09-06 14:10:41.574516108 +0000 UTC m=+790.837960832" Sep 06 14:10:44.550430 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:44.550395 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:10:47.571417 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:47.571390 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:10:57.687653 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.687568 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8"] Sep 06 14:10:57.692939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.692910 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.695245 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.695212 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5ec585-serving-cert\"" Sep 06 14:10:57.698830 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.698800 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8"] Sep 06 14:10:57.781796 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.781755 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.782049 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.781812 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.782049 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.781958 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872cd\" (UniqueName: \"kubernetes.io/projected/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-kube-api-access-872cd\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.782049 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.782031 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-metrics-certs\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.883079 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.883039 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.883271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.883094 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.883271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.883149 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872cd\" (UniqueName: \"kubernetes.io/projected/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-kube-api-access-872cd\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.883271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.883206 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-metrics-certs\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.883457 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:57.883288 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-serving-cert: secret "maas-api-e2e-inf-b-5ec585-serving-cert" not found Sep 06 14:10:57.883457 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:10:57.883391 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls podName:79a7bfb9-a7cb-467b-aaf6-165d26cb3a55 nodeName:}" failed. No retries permitted until 2026-09-06 14:10:58.383367707 +0000 UTC m=+807.646812431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls") pod "maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" (UID: "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55") : secret "maas-api-e2e-inf-b-5ec585-serving-cert" not found Sep 06 14:10:57.883822 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.883795 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.885737 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.885709 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-metrics-certs\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:57.893685 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:57.893664 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-872cd\" (UniqueName: \"kubernetes.io/projected/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-kube-api-access-872cd\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:58.388410 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.388376 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:58.391179 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.391147 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:58.391320 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.391220 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc"] Sep 06 14:10:58.395163 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.395144 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.397512 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.397491 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ec585\"" Sep 06 14:10:58.397693 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.397681 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ec585-dockercfg-k5d5v\"" Sep 06 14:10:58.404551 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.404516 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc"] Sep 06 14:10:58.489634 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.489600 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3713bde-3e66-4a70-8a20-c2d941c06548-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.489821 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.489664 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmff4\" (UniqueName: \"kubernetes.io/projected/c3713bde-3e66-4a70-8a20-c2d941c06548-kube-api-access-hmff4\") pod \"payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.590654 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.590608 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmff4\" (UniqueName: \"kubernetes.io/projected/c3713bde-3e66-4a70-8a20-c2d941c06548-kube-api-access-hmff4\") pod \"payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.590831 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.590730 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3713bde-3e66-4a70-8a20-c2d941c06548-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.591481 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.591461 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3713bde-3e66-4a70-8a20-c2d941c06548-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.597753 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.597723 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmff4\" (UniqueName: \"kubernetes.io/projected/c3713bde-3e66-4a70-8a20-c2d941c06548-kube-api-access-hmff4\") pod \"payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.607598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.607571 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:58.712639 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.712606 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:58.890282 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.890257 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8"] Sep 06 14:10:58.891872 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:58.891842 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a7bfb9_a7cb_467b_aaf6_165d26cb3a55.slice/crio-2d45647bccceeaf1170db109afa4aef62d78124bf5631c53baa1d9e39efde22c WatchSource:0}: Error finding container 2d45647bccceeaf1170db109afa4aef62d78124bf5631c53baa1d9e39efde22c: Status 404 returned error can't find the container with id 2d45647bccceeaf1170db109afa4aef62d78124bf5631c53baa1d9e39efde22c Sep 06 14:10:58.993348 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:58.993319 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc"] Sep 06 14:10:58.995835 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:10:58.995805 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3713bde_3e66_4a70_8a20_c2d941c06548.slice/crio-ffba62dda20de30b889e16b42f4952c2ffe3ec7559dc2ec36454c885639fe65b WatchSource:0}: Error finding container ffba62dda20de30b889e16b42f4952c2ffe3ec7559dc2ec36454c885639fe65b: Status 404 returned error can't find the container with id ffba62dda20de30b889e16b42f4952c2ffe3ec7559dc2ec36454c885639fe65b Sep 06 14:10:59.686103 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.686064 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" event={"ID":"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55","Type":"ContainerStarted","Data":"fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a"} Sep 06 14:10:59.686329 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.686128 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" event={"ID":"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55","Type":"ContainerStarted","Data":"2d45647bccceeaf1170db109afa4aef62d78124bf5631c53baa1d9e39efde22c"} Sep 06 14:10:59.686329 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.686274 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:10:59.697609 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.697560 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" event={"ID":"c3713bde-3e66-4a70-8a20-c2d941c06548","Type":"ContainerStarted","Data":"7c0855a9345eca207350b57516f45ac80fb7762b647b9c5d62892ab4e3b6e46b"} Sep 06 14:10:59.697802 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.697670 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" event={"ID":"c3713bde-3e66-4a70-8a20-c2d941c06548","Type":"ContainerStarted","Data":"ffba62dda20de30b889e16b42f4952c2ffe3ec7559dc2ec36454c885639fe65b"} Sep 06 14:10:59.697802 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.697705 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:10:59.704323 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.704256 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" podStartSLOduration=2.264786992 podStartE2EDuration="2.704236536s" totalImagesPullingTime="439.449544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:59.701703894 +0000 UTC m=+808.965148614" watchObservedRunningTime="2026-09-06 14:10:59.704236536 +0000 UTC m=+808.967681339" Sep 06 14:10:59.722168 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:10:59.722091 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" podStartSLOduration=1.722065793 podStartE2EDuration="1.722065793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:59.717336175 +0000 UTC m=+808.980780900" watchObservedRunningTime="2026-09-06 14:10:59.722065793 +0000 UTC m=+808.985510518" Sep 06 14:11:05.712959 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:05.712925 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:11:10.709264 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:10.709230 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:11:17.817587 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:17.817551 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n"] Sep 06 14:11:17.818103 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:17.817778 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" podUID="1f67d6e0-a5cb-496d-8717-7d2f29c8326a" containerName="maas-api" containerID="cri-o://d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a" gracePeriod=30 Sep 06 14:11:18.084035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.083965 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:11:18.180051 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.180009 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-openshift-service-ca\") pod \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " Sep 06 14:11:18.180249 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.180113 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-metrics-certs\") pod \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " Sep 06 14:11:18.180249 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.180142 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtft5\" (UniqueName: \"kubernetes.io/projected/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-kube-api-access-qtft5\") pod \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " Sep 06 14:11:18.180373 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.180249 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-maas-api-tls\") pod \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\" (UID: \"1f67d6e0-a5cb-496d-8717-7d2f29c8326a\") " Sep 06 14:11:18.180450 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.180419 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f67d6e0-a5cb-496d-8717-7d2f29c8326a" (UID: "1f67d6e0-a5cb-496d-8717-7d2f29c8326a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:18.180577 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.180561 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.182400 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.182372 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f67d6e0-a5cb-496d-8717-7d2f29c8326a" (UID: "1f67d6e0-a5cb-496d-8717-7d2f29c8326a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:11:18.182400 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.182390 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f67d6e0-a5cb-496d-8717-7d2f29c8326a" (UID: "1f67d6e0-a5cb-496d-8717-7d2f29c8326a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:11:18.182561 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.182477 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-kube-api-access-qtft5" (OuterVolumeSpecName: "kube-api-access-qtft5") pod "1f67d6e0-a5cb-496d-8717-7d2f29c8326a" (UID: "1f67d6e0-a5cb-496d-8717-7d2f29c8326a"). InnerVolumeSpecName "kube-api-access-qtft5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:18.281343 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.281307 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.281343 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.281341 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtft5\" (UniqueName: \"kubernetes.io/projected/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-kube-api-access-qtft5\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.281343 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.281351 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f67d6e0-a5cb-496d-8717-7d2f29c8326a-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:18.826503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.826467 2550 generic.go:356] "Generic (PLEG): container finished" podID="1f67d6e0-a5cb-496d-8717-7d2f29c8326a" containerID="d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a" exitCode=0 Sep 06 14:11:18.827040 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.826515 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" Sep 06 14:11:18.827040 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.826550 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" event={"ID":"1f67d6e0-a5cb-496d-8717-7d2f29c8326a","Type":"ContainerDied","Data":"d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a"} Sep 06 14:11:18.827040 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.826605 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n" event={"ID":"1f67d6e0-a5cb-496d-8717-7d2f29c8326a","Type":"ContainerDied","Data":"149706ead74d751010c35732dc2cf91d73e9ccd16a0d7d66b047d7284608f683"} Sep 06 14:11:18.827040 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.826627 2550 scope.go:117] "RemoveContainer" containerID="d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a" Sep 06 14:11:18.836511 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.836491 2550 scope.go:117] "RemoveContainer" containerID="d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a" Sep 06 14:11:18.836825 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:11:18.836804 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a\": container with ID starting with d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a not found: ID does not exist" containerID="d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a" Sep 06 14:11:18.836910 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.836839 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a"} err="failed to get container status \"d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a\": rpc error: code = NotFound desc = could not find container \"d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a\": container with ID starting with d4522874a741533041c0f8b9e19ea1472809c0854d890702fcd1fe7437347f0a not found: ID does not exist" Sep 06 14:11:18.848749 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.848719 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n"] Sep 06 14:11:18.852634 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:18.852605 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84753aa3-57d7755c5b-6928n"] Sep 06 14:11:19.236990 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:19.236954 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f67d6e0-a5cb-496d-8717-7d2f29c8326a" path="/var/lib/kubelet/pods/1f67d6e0-a5cb-496d-8717-7d2f29c8326a/volumes" Sep 06 14:11:21.787428 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.787391 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw"] Sep 06 14:11:21.788713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.788682 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f67d6e0-a5cb-496d-8717-7d2f29c8326a" containerName="maas-api" Sep 06 14:11:21.788713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.788710 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f67d6e0-a5cb-496d-8717-7d2f29c8326a" containerName="maas-api" Sep 06 14:11:21.788847 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.788826 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f67d6e0-a5cb-496d-8717-7d2f29c8326a" containerName="maas-api" Sep 06 14:11:21.792812 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.792788 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:21.795118 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.795096 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c0e08d9a-serving-cert\"" Sep 06 14:11:21.799134 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.799108 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw"] Sep 06 14:11:21.917978 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.917942 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:21.918177 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.918057 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca89d-5203-483d-950e-f6d8330f3837-openshift-service-ca\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:21.918177 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.918097 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5tz2\" (UniqueName: \"kubernetes.io/projected/8bcca89d-5203-483d-950e-f6d8330f3837-kube-api-access-t5tz2\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:21.918177 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:21.918132 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-metrics-certs\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.019298 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.019249 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.019513 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.019330 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca89d-5203-483d-950e-f6d8330f3837-openshift-service-ca\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.019513 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.019368 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5tz2\" (UniqueName: \"kubernetes.io/projected/8bcca89d-5203-483d-950e-f6d8330f3837-kube-api-access-t5tz2\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.019513 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.019405 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-metrics-certs\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.019513 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:11:22.019427 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-serving-cert: secret "maas-api-e2e-migrate-c0e08d9a-serving-cert" not found Sep 06 14:11:22.019513 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:11:22.019496 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls podName:8bcca89d-5203-483d-950e-f6d8330f3837 nodeName:}" failed. No retries permitted until 2026-09-06 14:11:22.51947618 +0000 UTC m=+831.782920905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls") pod "maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" (UID: "8bcca89d-5203-483d-950e-f6d8330f3837") : secret "maas-api-e2e-migrate-c0e08d9a-serving-cert" not found Sep 06 14:11:22.020009 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.019987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca89d-5203-483d-950e-f6d8330f3837-openshift-service-ca\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.021823 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.021805 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-metrics-certs\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.033830 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.033805 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5tz2\" (UniqueName: \"kubernetes.io/projected/8bcca89d-5203-483d-950e-f6d8330f3837-kube-api-access-t5tz2\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.524164 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.524116 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.526630 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.526599 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls\") pod \"maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.710037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.709994 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:22.985082 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:22.985054 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw"] Sep 06 14:11:22.987015 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:11:22.986985 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcca89d_5203_483d_950e_f6d8330f3837.slice/crio-6c3c71228a40a64c4273997c1be530f379d6c1f5db40128b2721f8965142800f WatchSource:0}: Error finding container 6c3c71228a40a64c4273997c1be530f379d6c1f5db40128b2721f8965142800f: Status 404 returned error can't find the container with id 6c3c71228a40a64c4273997c1be530f379d6c1f5db40128b2721f8965142800f Sep 06 14:11:23.862383 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:23.862343 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" event={"ID":"8bcca89d-5203-483d-950e-f6d8330f3837","Type":"ContainerStarted","Data":"645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f"} Sep 06 14:11:23.862558 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:23.862410 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:23.862558 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:23.862437 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" event={"ID":"8bcca89d-5203-483d-950e-f6d8330f3837","Type":"ContainerStarted","Data":"6c3c71228a40a64c4273997c1be530f379d6c1f5db40128b2721f8965142800f"} Sep 06 14:11:23.877731 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:23.877670 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" podStartSLOduration=2.512149888 podStartE2EDuration="2.877652401s" totalImagesPullingTime="365.502513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:23.876187259 +0000 UTC m=+833.139631985" watchObservedRunningTime="2026-09-06 14:11:23.877652401 +0000 UTC m=+833.141097171" Sep 06 14:11:29.874916 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:29.874872 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:52.292547 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.292507 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw"] Sep 06 14:11:52.293174 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.292825 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" podUID="8bcca89d-5203-483d-950e-f6d8330f3837" containerName="maas-api" containerID="cri-o://645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f" gracePeriod=30 Sep 06 14:11:52.569660 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.569630 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:52.707699 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.707664 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5tz2\" (UniqueName: \"kubernetes.io/projected/8bcca89d-5203-483d-950e-f6d8330f3837-kube-api-access-t5tz2\") pod \"8bcca89d-5203-483d-950e-f6d8330f3837\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " Sep 06 14:11:52.707887 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.707707 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls\") pod \"8bcca89d-5203-483d-950e-f6d8330f3837\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " Sep 06 14:11:52.707887 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.707765 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca89d-5203-483d-950e-f6d8330f3837-openshift-service-ca\") pod \"8bcca89d-5203-483d-950e-f6d8330f3837\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " Sep 06 14:11:52.707887 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.707790 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-metrics-certs\") pod \"8bcca89d-5203-483d-950e-f6d8330f3837\" (UID: \"8bcca89d-5203-483d-950e-f6d8330f3837\") " Sep 06 14:11:52.708223 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.708190 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcca89d-5203-483d-950e-f6d8330f3837-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bcca89d-5203-483d-950e-f6d8330f3837" (UID: "8bcca89d-5203-483d-950e-f6d8330f3837"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:11:52.709939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.709901 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bcca89d-5203-483d-950e-f6d8330f3837" (UID: "8bcca89d-5203-483d-950e-f6d8330f3837"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:11:52.710354 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.710329 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bcca89d-5203-483d-950e-f6d8330f3837" (UID: "8bcca89d-5203-483d-950e-f6d8330f3837"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:11:52.710428 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.710349 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcca89d-5203-483d-950e-f6d8330f3837-kube-api-access-t5tz2" (OuterVolumeSpecName: "kube-api-access-t5tz2") pod "8bcca89d-5203-483d-950e-f6d8330f3837" (UID: "8bcca89d-5203-483d-950e-f6d8330f3837"). InnerVolumeSpecName "kube-api-access-t5tz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:11:52.808720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.808624 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5tz2\" (UniqueName: \"kubernetes.io/projected/8bcca89d-5203-483d-950e-f6d8330f3837-kube-api-access-t5tz2\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:52.808720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.808662 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:52.808720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.808671 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcca89d-5203-483d-950e-f6d8330f3837-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:52.808720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:52.808680 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcca89d-5203-483d-950e-f6d8330f3837-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:11:53.056596 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.056563 2550 generic.go:356] "Generic (PLEG): container finished" podID="8bcca89d-5203-483d-950e-f6d8330f3837" containerID="645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f" exitCode=0 Sep 06 14:11:53.056774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.056613 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" Sep 06 14:11:53.056774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.056650 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" event={"ID":"8bcca89d-5203-483d-950e-f6d8330f3837","Type":"ContainerDied","Data":"645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f"} Sep 06 14:11:53.056774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.056705 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw" event={"ID":"8bcca89d-5203-483d-950e-f6d8330f3837","Type":"ContainerDied","Data":"6c3c71228a40a64c4273997c1be530f379d6c1f5db40128b2721f8965142800f"} Sep 06 14:11:53.056774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.056727 2550 scope.go:117] "RemoveContainer" containerID="645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f" Sep 06 14:11:53.067778 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.067750 2550 scope.go:117] "RemoveContainer" containerID="645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f" Sep 06 14:11:53.068144 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:11:53.068123 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f\": container with ID starting with 645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f not found: ID does not exist" containerID="645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f" Sep 06 14:11:53.068216 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.068151 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f"} err="failed to get container status \"645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f\": rpc error: code = NotFound desc = could not find container \"645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f\": container with ID starting with 645819434f27a8c18b4b8298af638d5d4314eb509fc8dfe5e7117584f8b6ce6f not found: ID does not exist" Sep 06 14:11:53.078442 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.078408 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw"] Sep 06 14:11:53.081416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.081387 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c0e08d9a-7d7fc4dbbf-26hjw"] Sep 06 14:11:53.236969 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:53.236933 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcca89d-5203-483d-950e-f6d8330f3837" path="/var/lib/kubelet/pods/8bcca89d-5203-483d-950e-f6d8330f3837/volumes" Sep 06 14:11:56.719875 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.719829 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l"] Sep 06 14:11:56.720880 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.720851 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcca89d-5203-483d-950e-f6d8330f3837" containerName="maas-api" Sep 06 14:11:56.721018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.720884 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcca89d-5203-483d-950e-f6d8330f3837" containerName="maas-api" Sep 06 14:11:56.721193 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.721174 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcca89d-5203-483d-950e-f6d8330f3837" containerName="maas-api" Sep 06 14:11:56.727095 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.727064 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.730744 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.730716 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d954782c-serving-cert\"" Sep 06 14:11:56.731037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.730817 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l"] Sep 06 14:11:56.847819 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.847773 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-metrics-certs\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.848005 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.847832 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f62bc-19ad-4308-b408-b50b027fefcb-openshift-service-ca\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.848005 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.847954 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmz5d\" (UniqueName: \"kubernetes.io/projected/126f62bc-19ad-4308-b408-b50b027fefcb-kube-api-access-qmz5d\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.848117 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.848008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.948847 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.948800 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.949069 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.948866 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-metrics-certs\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.949069 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.948922 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f62bc-19ad-4308-b408-b50b027fefcb-openshift-service-ca\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.949069 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.948972 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmz5d\" (UniqueName: \"kubernetes.io/projected/126f62bc-19ad-4308-b408-b50b027fefcb-kube-api-access-qmz5d\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.949069 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:11:56.949045 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-serving-cert: secret "maas-api-e2e-ait-d954782c-serving-cert" not found Sep 06 14:11:56.949309 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:11:56.949136 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls podName:126f62bc-19ad-4308-b408-b50b027fefcb nodeName:}" failed. No retries permitted until 2026-09-06 14:11:57.449113476 +0000 UTC m=+866.712558179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls") pod "maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" (UID: "126f62bc-19ad-4308-b408-b50b027fefcb") : secret "maas-api-e2e-ait-d954782c-serving-cert" not found Sep 06 14:11:56.950003 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.949839 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f62bc-19ad-4308-b408-b50b027fefcb-openshift-service-ca\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.951687 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.951663 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-metrics-certs\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:56.957512 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:56.957478 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmz5d\" (UniqueName: \"kubernetes.io/projected/126f62bc-19ad-4308-b408-b50b027fefcb-kube-api-access-qmz5d\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:57.453003 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:57.452964 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:57.455563 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:57.455537 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls\") pod \"maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:57.643821 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:57.643769 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:57.918539 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:57.918511 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l"] Sep 06 14:11:57.919880 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:11:57.919856 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126f62bc_19ad_4308_b408_b50b027fefcb.slice/crio-f4b149c6a89347b5f9b24b555fe05137627712692ca53a7241db6a14e2fdcd27 WatchSource:0}: Error finding container f4b149c6a89347b5f9b24b555fe05137627712692ca53a7241db6a14e2fdcd27: Status 404 returned error can't find the container with id f4b149c6a89347b5f9b24b555fe05137627712692ca53a7241db6a14e2fdcd27 Sep 06 14:11:58.090050 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:58.090010 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" event={"ID":"126f62bc-19ad-4308-b408-b50b027fefcb","Type":"ContainerStarted","Data":"f4b149c6a89347b5f9b24b555fe05137627712692ca53a7241db6a14e2fdcd27"} Sep 06 14:11:59.098192 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:59.098154 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" event={"ID":"126f62bc-19ad-4308-b408-b50b027fefcb","Type":"ContainerStarted","Data":"8326a4b0a054cff06c167cec56ea30d74e43bcc69a6394c5b22cee63c8e972ea"} Sep 06 14:11:59.098642 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:59.098342 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:11:59.114756 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:11:59.114699 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" podStartSLOduration=2.749318106 podStartE2EDuration="3.114681466s" totalImagesPullingTime="365.36336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:59.113959862 +0000 UTC m=+868.377404586" watchObservedRunningTime="2026-09-06 14:11:59.114681466 +0000 UTC m=+868.378126189" Sep 06 14:12:05.110308 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:05.110278 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:12:27.160373 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.160327 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l"] Sep 06 14:12:27.160926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.160660 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" podUID="126f62bc-19ad-4308-b408-b50b027fefcb" containerName="maas-api" containerID="cri-o://8326a4b0a054cff06c167cec56ea30d74e43bcc69a6394c5b22cee63c8e972ea" gracePeriod=30 Sep 06 14:12:27.284280 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.284247 2550 generic.go:356] "Generic (PLEG): container finished" podID="126f62bc-19ad-4308-b408-b50b027fefcb" containerID="8326a4b0a054cff06c167cec56ea30d74e43bcc69a6394c5b22cee63c8e972ea" exitCode=0 Sep 06 14:12:27.284481 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.284319 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" event={"ID":"126f62bc-19ad-4308-b408-b50b027fefcb","Type":"ContainerDied","Data":"8326a4b0a054cff06c167cec56ea30d74e43bcc69a6394c5b22cee63c8e972ea"} Sep 06 14:12:27.427252 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.427224 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:12:27.535488 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.535454 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-metrics-certs\") pod \"126f62bc-19ad-4308-b408-b50b027fefcb\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " Sep 06 14:12:27.535488 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.535491 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmz5d\" (UniqueName: \"kubernetes.io/projected/126f62bc-19ad-4308-b408-b50b027fefcb-kube-api-access-qmz5d\") pod \"126f62bc-19ad-4308-b408-b50b027fefcb\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " Sep 06 14:12:27.535718 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.535568 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls\") pod \"126f62bc-19ad-4308-b408-b50b027fefcb\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " Sep 06 14:12:27.535718 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.535634 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f62bc-19ad-4308-b408-b50b027fefcb-openshift-service-ca\") pod \"126f62bc-19ad-4308-b408-b50b027fefcb\" (UID: \"126f62bc-19ad-4308-b408-b50b027fefcb\") " Sep 06 14:12:27.536153 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.536117 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126f62bc-19ad-4308-b408-b50b027fefcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "126f62bc-19ad-4308-b408-b50b027fefcb" (UID: "126f62bc-19ad-4308-b408-b50b027fefcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:27.539654 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.539557 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "126f62bc-19ad-4308-b408-b50b027fefcb" (UID: "126f62bc-19ad-4308-b408-b50b027fefcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:12:27.539803 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.539733 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126f62bc-19ad-4308-b408-b50b027fefcb-kube-api-access-qmz5d" (OuterVolumeSpecName: "kube-api-access-qmz5d") pod "126f62bc-19ad-4308-b408-b50b027fefcb" (UID: "126f62bc-19ad-4308-b408-b50b027fefcb"). InnerVolumeSpecName "kube-api-access-qmz5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:12:27.539885 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.539853 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "126f62bc-19ad-4308-b408-b50b027fefcb" (UID: "126f62bc-19ad-4308-b408-b50b027fefcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:12:27.636501 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.636463 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.636501 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.636502 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f62bc-19ad-4308-b408-b50b027fefcb-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.636501 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.636519 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f62bc-19ad-4308-b408-b50b027fefcb-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:27.636801 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:27.636534 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmz5d\" (UniqueName: \"kubernetes.io/projected/126f62bc-19ad-4308-b408-b50b027fefcb-kube-api-access-qmz5d\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:28.292277 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:28.292250 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" Sep 06 14:12:28.292685 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:28.292278 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l" event={"ID":"126f62bc-19ad-4308-b408-b50b027fefcb","Type":"ContainerDied","Data":"f4b149c6a89347b5f9b24b555fe05137627712692ca53a7241db6a14e2fdcd27"} Sep 06 14:12:28.292685 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:28.292330 2550 scope.go:117] "RemoveContainer" containerID="8326a4b0a054cff06c167cec56ea30d74e43bcc69a6394c5b22cee63c8e972ea" Sep 06 14:12:28.312992 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:28.312959 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l"] Sep 06 14:12:28.319000 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:28.318968 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d954782c-6f57b8c4f8-kzl2l"] Sep 06 14:12:29.237627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:29.237586 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="126f62bc-19ad-4308-b408-b50b027fefcb" path="/var/lib/kubelet/pods/126f62bc-19ad-4308-b408-b50b027fefcb/volumes" Sep 06 14:12:31.255000 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:31.254955 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:12:31.255672 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:31.255652 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:12:33.187430 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.187396 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt"] Sep 06 14:12:33.189825 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.187839 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126f62bc-19ad-4308-b408-b50b027fefcb" containerName="maas-api" Sep 06 14:12:33.189825 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.187852 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="126f62bc-19ad-4308-b408-b50b027fefcb" containerName="maas-api" Sep 06 14:12:33.189825 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.187941 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="126f62bc-19ad-4308-b408-b50b027fefcb" containerName="maas-api" Sep 06 14:12:33.190861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.190843 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.192970 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.192946 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b1ed35f1-serving-cert\"" Sep 06 14:12:33.202711 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.202679 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt"] Sep 06 14:12:33.288401 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.288362 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-metrics-certs\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.288594 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.288440 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.288594 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.288495 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3318d341-def1-4116-bfba-a5419c2ce51a-openshift-service-ca\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.288594 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.288577 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvbm\" (UniqueName: \"kubernetes.io/projected/3318d341-def1-4116-bfba-a5419c2ce51a-kube-api-access-snvbm\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.389775 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.389740 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-metrics-certs\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.389987 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.389795 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.389987 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.389845 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3318d341-def1-4116-bfba-a5419c2ce51a-openshift-service-ca\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.389987 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.389871 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvbm\" (UniqueName: \"kubernetes.io/projected/3318d341-def1-4116-bfba-a5419c2ce51a-kube-api-access-snvbm\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.390175 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:12:33.390008 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-serving-cert: secret "maas-api-e2e-derive-b1ed35f1-serving-cert" not found Sep 06 14:12:33.390175 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:12:33.390096 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls podName:3318d341-def1-4116-bfba-a5419c2ce51a nodeName:}" failed. No retries permitted until 2026-09-06 14:12:33.890072945 +0000 UTC m=+903.153517662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls") pod "maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" (UID: "3318d341-def1-4116-bfba-a5419c2ce51a") : secret "maas-api-e2e-derive-b1ed35f1-serving-cert" not found Sep 06 14:12:33.390646 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.390620 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3318d341-def1-4116-bfba-a5419c2ce51a-openshift-service-ca\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.392423 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.392395 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-metrics-certs\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.397614 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.397591 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvbm\" (UniqueName: \"kubernetes.io/projected/3318d341-def1-4116-bfba-a5419c2ce51a-kube-api-access-snvbm\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.895658 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.895620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:33.898212 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:33.898184 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls\") pod \"maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:34.104238 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:34.104202 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:34.385818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:34.385787 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt"] Sep 06 14:12:34.389365 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:12:34.389319 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3318d341_def1_4116_bfba_a5419c2ce51a.slice/crio-1ca141fd436a3f1553570899d6bca3718be8271760108bb92afdcd478c777904 WatchSource:0}: Error finding container 1ca141fd436a3f1553570899d6bca3718be8271760108bb92afdcd478c777904: Status 404 returned error can't find the container with id 1ca141fd436a3f1553570899d6bca3718be8271760108bb92afdcd478c777904 Sep 06 14:12:35.340709 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:35.340657 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" event={"ID":"3318d341-def1-4116-bfba-a5419c2ce51a","Type":"ContainerStarted","Data":"7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981"} Sep 06 14:12:35.340952 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:35.340735 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:35.340952 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:35.340758 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" event={"ID":"3318d341-def1-4116-bfba-a5419c2ce51a","Type":"ContainerStarted","Data":"1ca141fd436a3f1553570899d6bca3718be8271760108bb92afdcd478c777904"} Sep 06 14:12:35.357664 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:35.357601 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" podStartSLOduration=1.9218786749999999 podStartE2EDuration="2.357579693s" totalImagesPullingTime="435.701018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:35.355173279 +0000 UTC m=+904.618618006" watchObservedRunningTime="2026-09-06 14:12:35.357579693 +0000 UTC m=+904.621024416" Sep 06 14:12:41.352248 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:41.352212 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:12:41.789547 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:41.789511 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25"] Sep 06 14:12:41.789784 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:41.789756 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" podUID="ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" containerName="maas-api" containerID="cri-o://a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235" gracePeriod=30 Sep 06 14:12:42.097246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.097217 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:12:42.174956 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.174917 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-metrics-certs\") pod \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " Sep 06 14:12:42.175150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.175042 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-maas-api-tls\") pod \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " Sep 06 14:12:42.175150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.175078 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j4vw\" (UniqueName: \"kubernetes.io/projected/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-kube-api-access-4j4vw\") pod \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " Sep 06 14:12:42.175150 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.175102 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-openshift-service-ca\") pod \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\" (UID: \"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e\") " Sep 06 14:12:42.175535 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.175500 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" (UID: "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:12:42.177315 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.177283 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" (UID: "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:12:42.177452 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.177332 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" (UID: "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:12:42.177452 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.177338 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-kube-api-access-4j4vw" (OuterVolumeSpecName: "kube-api-access-4j4vw") pod "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" (UID: "ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e"). InnerVolumeSpecName "kube-api-access-4j4vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:12:42.277358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.277052 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.277358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.277088 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4vw\" (UniqueName: \"kubernetes.io/projected/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-kube-api-access-4j4vw\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.277358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.277108 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.277358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.277121 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:12:42.386504 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.386403 2550 generic.go:356] "Generic (PLEG): container finished" podID="ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" containerID="a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235" exitCode=0 Sep 06 14:12:42.386504 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.386457 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" Sep 06 14:12:42.387060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.386491 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" event={"ID":"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e","Type":"ContainerDied","Data":"a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235"} Sep 06 14:12:42.387060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.386562 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25" event={"ID":"ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e","Type":"ContainerDied","Data":"7e8d88d857548622149fc3367962b52d1c8021379ca1b2051385a0be564ff84c"} Sep 06 14:12:42.387060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.386594 2550 scope.go:117] "RemoveContainer" containerID="a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235" Sep 06 14:12:42.396880 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.396860 2550 scope.go:117] "RemoveContainer" containerID="a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235" Sep 06 14:12:42.397288 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:12:42.397258 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235\": container with ID starting with a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235 not found: ID does not exist" containerID="a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235" Sep 06 14:12:42.397345 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.397301 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235"} err="failed to get container status \"a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235\": rpc error: code = NotFound desc = could not find container \"a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235\": container with ID starting with a2976eeda791619edb26f6855b9412ecf1763ed8cbe5ad78ea2a0d05675f6235 not found: ID does not exist" Sep 06 14:12:42.408981 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.408946 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25"] Sep 06 14:12:42.412823 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:42.412775 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1482ae-6ccbcfc586-hdc25"] Sep 06 14:12:43.236828 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:12:43.236788 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" path="/var/lib/kubelet/pods/ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e/volumes" Sep 06 14:13:03.715468 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:03.715428 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt"] Sep 06 14:13:03.716134 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:03.716073 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" podUID="3318d341-def1-4116-bfba-a5419c2ce51a" containerName="maas-api" containerID="cri-o://7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981" gracePeriod=30 Sep 06 14:13:03.995679 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:03.995647 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:13:04.186972 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.186934 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3318d341-def1-4116-bfba-a5419c2ce51a-openshift-service-ca\") pod \"3318d341-def1-4116-bfba-a5419c2ce51a\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " Sep 06 14:13:04.187157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.186992 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls\") pod \"3318d341-def1-4116-bfba-a5419c2ce51a\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " Sep 06 14:13:04.187157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.187022 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snvbm\" (UniqueName: \"kubernetes.io/projected/3318d341-def1-4116-bfba-a5419c2ce51a-kube-api-access-snvbm\") pod \"3318d341-def1-4116-bfba-a5419c2ce51a\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " Sep 06 14:13:04.187157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.187105 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-metrics-certs\") pod \"3318d341-def1-4116-bfba-a5419c2ce51a\" (UID: \"3318d341-def1-4116-bfba-a5419c2ce51a\") " Sep 06 14:13:04.187391 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.187365 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3318d341-def1-4116-bfba-a5419c2ce51a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3318d341-def1-4116-bfba-a5419c2ce51a" (UID: "3318d341-def1-4116-bfba-a5419c2ce51a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:13:04.189211 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.189183 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3318d341-def1-4116-bfba-a5419c2ce51a" (UID: "3318d341-def1-4116-bfba-a5419c2ce51a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:13:04.189717 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.189691 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3318d341-def1-4116-bfba-a5419c2ce51a-kube-api-access-snvbm" (OuterVolumeSpecName: "kube-api-access-snvbm") pod "3318d341-def1-4116-bfba-a5419c2ce51a" (UID: "3318d341-def1-4116-bfba-a5419c2ce51a"). InnerVolumeSpecName "kube-api-access-snvbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:13:04.189825 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.189803 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3318d341-def1-4116-bfba-a5419c2ce51a" (UID: "3318d341-def1-4116-bfba-a5419c2ce51a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:13:04.288882 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.288793 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.288882 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.288826 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3318d341-def1-4116-bfba-a5419c2ce51a-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.288882 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.288836 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3318d341-def1-4116-bfba-a5419c2ce51a-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.288882 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.288844 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvbm\" (UniqueName: \"kubernetes.io/projected/3318d341-def1-4116-bfba-a5419c2ce51a-kube-api-access-snvbm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:13:04.538353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.538304 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" Sep 06 14:13:04.538353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.538341 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" event={"ID":"3318d341-def1-4116-bfba-a5419c2ce51a","Type":"ContainerDied","Data":"7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981"} Sep 06 14:13:04.538632 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.538305 2550 generic.go:356] "Generic (PLEG): container finished" podID="3318d341-def1-4116-bfba-a5419c2ce51a" containerID="7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981" exitCode=0 Sep 06 14:13:04.538632 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.538412 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt" event={"ID":"3318d341-def1-4116-bfba-a5419c2ce51a","Type":"ContainerDied","Data":"1ca141fd436a3f1553570899d6bca3718be8271760108bb92afdcd478c777904"} Sep 06 14:13:04.538632 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.538442 2550 scope.go:117] "RemoveContainer" containerID="7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981" Sep 06 14:13:04.550160 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.550139 2550 scope.go:117] "RemoveContainer" containerID="7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981" Sep 06 14:13:04.550518 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:13:04.550499 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981\": container with ID starting with 7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981 not found: ID does not exist" containerID="7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981" Sep 06 14:13:04.550587 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.550529 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981"} err="failed to get container status \"7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981\": rpc error: code = NotFound desc = could not find container \"7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981\": container with ID starting with 7abc2c0f550eb020d2ab3c590f3c88442c0bc70b5ea7f4c3bf52a6d5845cc981 not found: ID does not exist" Sep 06 14:13:04.561917 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.561865 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt"] Sep 06 14:13:04.565358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:04.565325 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b1ed35f1-7bc9f55dd8-bpkjt"] Sep 06 14:13:05.236325 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:05.236285 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3318d341-def1-4116-bfba-a5419c2ce51a" path="/var/lib/kubelet/pods/3318d341-def1-4116-bfba-a5419c2ce51a/volumes" Sep 06 14:13:28.920514 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.920469 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq"] Sep 06 14:13:28.921039 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.920946 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3318d341-def1-4116-bfba-a5419c2ce51a" containerName="maas-api" Sep 06 14:13:28.921039 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.920965 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3318d341-def1-4116-bfba-a5419c2ce51a" containerName="maas-api" Sep 06 14:13:28.921039 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.920982 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" containerName="maas-api" Sep 06 14:13:28.921039 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.920988 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" containerName="maas-api" Sep 06 14:13:28.921269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.921087 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9c575f-a4e6-47c7-a36c-d5d6e3857c7e" containerName="maas-api" Sep 06 14:13:28.921269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.921100 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3318d341-def1-4116-bfba-a5419c2ce51a" containerName="maas-api" Sep 06 14:13:28.925049 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.924613 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:28.927666 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.927637 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-dfca8b-serving-cert\"" Sep 06 14:13:28.936257 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:28.936228 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq"] Sep 06 14:13:29.004980 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.004937 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxk9w\" (UniqueName: \"kubernetes.io/projected/c75417b8-c211-4b0d-b7d1-a21e9151e317-kube-api-access-gxk9w\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.005163 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.005021 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-maas-api-tls\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.005163 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.005046 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75417b8-c211-4b0d-b7d1-a21e9151e317-openshift-service-ca\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.005163 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.005074 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-metrics-certs\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.106480 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.106441 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-maas-api-tls\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.106480 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.106485 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75417b8-c211-4b0d-b7d1-a21e9151e317-openshift-service-ca\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.106702 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.106586 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-metrics-certs\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.106702 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.106634 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxk9w\" (UniqueName: \"kubernetes.io/projected/c75417b8-c211-4b0d-b7d1-a21e9151e317-kube-api-access-gxk9w\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.107395 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.107370 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75417b8-c211-4b0d-b7d1-a21e9151e317-openshift-service-ca\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.109019 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.108997 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-maas-api-tls\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.109115 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.109041 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-metrics-certs\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.114035 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.114013 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxk9w\" (UniqueName: \"kubernetes.io/projected/c75417b8-c211-4b0d-b7d1-a21e9151e317-kube-api-access-gxk9w\") pod \"maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.238433 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.238324 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:29.530211 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.530173 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:13:29.533588 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.533561 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq"] Sep 06 14:13:29.709743 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:29.709703 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" event={"ID":"c75417b8-c211-4b0d-b7d1-a21e9151e317","Type":"ContainerStarted","Data":"b8d2ea0df661d30032a7fd08e754050e0d467e407ecc2bb0300459264d27a5f5"} Sep 06 14:13:30.717554 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:30.717517 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" event={"ID":"c75417b8-c211-4b0d-b7d1-a21e9151e317","Type":"ContainerStarted","Data":"fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947"} Sep 06 14:13:30.717945 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:30.717686 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:13:30.733247 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:30.733193 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" podStartSLOduration=2.25874937 podStartE2EDuration="2.733173306s" totalImagesPullingTime="474.423936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:13:30.732364313 +0000 UTC m=+959.995809038" watchObservedRunningTime="2026-09-06 14:13:30.733173306 +0000 UTC m=+959.996618031" Sep 06 14:13:36.729853 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:13:36.729824 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:14:16.686967 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:16.686926 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq"] Sep 06 14:14:16.687526 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:16.687172 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" podUID="c75417b8-c211-4b0d-b7d1-a21e9151e317" containerName="maas-api" containerID="cri-o://fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947" gracePeriod=30 Sep 06 14:14:16.725233 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:16.725184 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" podUID="c75417b8-c211-4b0d-b7d1-a21e9151e317" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.72:8443/health\": dial tcp 10.132.0.72:8443: connect: connection refused" Sep 06 14:14:16.954732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:16.954654 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:14:17.003190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.003157 2550 generic.go:356] "Generic (PLEG): container finished" podID="c75417b8-c211-4b0d-b7d1-a21e9151e317" containerID="fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947" exitCode=0 Sep 06 14:14:17.003369 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.003203 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" Sep 06 14:14:17.003369 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.003248 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" event={"ID":"c75417b8-c211-4b0d-b7d1-a21e9151e317","Type":"ContainerDied","Data":"fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947"} Sep 06 14:14:17.003369 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.003297 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq" event={"ID":"c75417b8-c211-4b0d-b7d1-a21e9151e317","Type":"ContainerDied","Data":"b8d2ea0df661d30032a7fd08e754050e0d467e407ecc2bb0300459264d27a5f5"} Sep 06 14:14:17.003369 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.003324 2550 scope.go:117] "RemoveContainer" containerID="fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947" Sep 06 14:14:17.016079 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.016053 2550 scope.go:117] "RemoveContainer" containerID="fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947" Sep 06 14:14:17.016455 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:17.016429 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947\": container with ID starting with fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947 not found: ID does not exist" containerID="fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947" Sep 06 14:14:17.016570 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.016462 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947"} err="failed to get container status \"fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947\": rpc error: code = NotFound desc = could not find container \"fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947\": container with ID starting with fc12e8f93f5507d71cd3f09816c85f913b68bf386704a9700fb7814dedb16947 not found: ID does not exist" Sep 06 14:14:17.062080 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.062047 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxk9w\" (UniqueName: \"kubernetes.io/projected/c75417b8-c211-4b0d-b7d1-a21e9151e317-kube-api-access-gxk9w\") pod \"c75417b8-c211-4b0d-b7d1-a21e9151e317\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " Sep 06 14:14:17.062252 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.062088 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-metrics-certs\") pod \"c75417b8-c211-4b0d-b7d1-a21e9151e317\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " Sep 06 14:14:17.062252 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.062132 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-maas-api-tls\") pod \"c75417b8-c211-4b0d-b7d1-a21e9151e317\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " Sep 06 14:14:17.062252 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.062153 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75417b8-c211-4b0d-b7d1-a21e9151e317-openshift-service-ca\") pod \"c75417b8-c211-4b0d-b7d1-a21e9151e317\" (UID: \"c75417b8-c211-4b0d-b7d1-a21e9151e317\") " Sep 06 14:14:17.062646 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.062612 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75417b8-c211-4b0d-b7d1-a21e9151e317-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c75417b8-c211-4b0d-b7d1-a21e9151e317" (UID: "c75417b8-c211-4b0d-b7d1-a21e9151e317"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:17.064328 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.064295 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c75417b8-c211-4b0d-b7d1-a21e9151e317" (UID: "c75417b8-c211-4b0d-b7d1-a21e9151e317"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:14:17.064440 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.064334 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75417b8-c211-4b0d-b7d1-a21e9151e317-kube-api-access-gxk9w" (OuterVolumeSpecName: "kube-api-access-gxk9w") pod "c75417b8-c211-4b0d-b7d1-a21e9151e317" (UID: "c75417b8-c211-4b0d-b7d1-a21e9151e317"). InnerVolumeSpecName "kube-api-access-gxk9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:17.064440 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.064349 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c75417b8-c211-4b0d-b7d1-a21e9151e317" (UID: "c75417b8-c211-4b0d-b7d1-a21e9151e317"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:14:17.162844 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.162806 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxk9w\" (UniqueName: \"kubernetes.io/projected/c75417b8-c211-4b0d-b7d1-a21e9151e317-kube-api-access-gxk9w\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.162844 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.162837 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.162844 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.162847 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75417b8-c211-4b0d-b7d1-a21e9151e317-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.163133 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.162858 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75417b8-c211-4b0d-b7d1-a21e9151e317-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:17.319318 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.319280 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq"] Sep 06 14:14:17.325314 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:17.325289 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dfca8b-54df4c44b4-qz7hq"] Sep 06 14:14:19.237125 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:19.237090 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c75417b8-c211-4b0d-b7d1-a21e9151e317" path="/var/lib/kubelet/pods/c75417b8-c211-4b0d-b7d1-a21e9151e317/volumes" Sep 06 14:14:30.468271 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:30.468231 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595"] Sep 06 14:14:30.468757 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:30.468523 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" podUID="9045e982-5e27-4dd8-abfa-73fbce19c750" containerName="maas-api" containerID="cri-o://7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205" gracePeriod=30 Sep 06 14:14:30.934673 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:30.934640 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:14:31.095269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.095172 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls\") pod \"9045e982-5e27-4dd8-abfa-73fbce19c750\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " Sep 06 14:14:31.095269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.095219 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xvl\" (UniqueName: \"kubernetes.io/projected/9045e982-5e27-4dd8-abfa-73fbce19c750-kube-api-access-m8xvl\") pod \"9045e982-5e27-4dd8-abfa-73fbce19c750\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " Sep 06 14:14:31.095269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.095247 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045e982-5e27-4dd8-abfa-73fbce19c750-openshift-service-ca\") pod \"9045e982-5e27-4dd8-abfa-73fbce19c750\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " Sep 06 14:14:31.095269 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.095273 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-metrics-certs\") pod \"9045e982-5e27-4dd8-abfa-73fbce19c750\" (UID: \"9045e982-5e27-4dd8-abfa-73fbce19c750\") " Sep 06 14:14:31.095651 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.095624 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9045e982-5e27-4dd8-abfa-73fbce19c750-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9045e982-5e27-4dd8-abfa-73fbce19c750" (UID: "9045e982-5e27-4dd8-abfa-73fbce19c750"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:31.097669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.097621 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9045e982-5e27-4dd8-abfa-73fbce19c750-kube-api-access-m8xvl" (OuterVolumeSpecName: "kube-api-access-m8xvl") pod "9045e982-5e27-4dd8-abfa-73fbce19c750" (UID: "9045e982-5e27-4dd8-abfa-73fbce19c750"). InnerVolumeSpecName "kube-api-access-m8xvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:31.097840 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.097624 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9045e982-5e27-4dd8-abfa-73fbce19c750" (UID: "9045e982-5e27-4dd8-abfa-73fbce19c750"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:14:31.097840 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.097693 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9045e982-5e27-4dd8-abfa-73fbce19c750" (UID: "9045e982-5e27-4dd8-abfa-73fbce19c750"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:14:31.098476 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.098451 2550 generic.go:356] "Generic (PLEG): container finished" podID="9045e982-5e27-4dd8-abfa-73fbce19c750" containerID="7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205" exitCode=0 Sep 06 14:14:31.098533 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.098507 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" event={"ID":"9045e982-5e27-4dd8-abfa-73fbce19c750","Type":"ContainerDied","Data":"7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205"} Sep 06 14:14:31.098533 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.098520 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" Sep 06 14:14:31.098619 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.098551 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595" event={"ID":"9045e982-5e27-4dd8-abfa-73fbce19c750","Type":"ContainerDied","Data":"3fbcc3b5b2140befd8675eb2c1e980edc23bd6a0e066d45fa00b8c8ec5973fb8"} Sep 06 14:14:31.098619 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.098572 2550 scope.go:117] "RemoveContainer" containerID="7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205" Sep 06 14:14:31.122868 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.122846 2550 scope.go:117] "RemoveContainer" containerID="7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205" Sep 06 14:14:31.123289 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:31.123249 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205\": container with ID starting with 7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205 not found: ID does not exist" containerID="7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205" Sep 06 14:14:31.123369 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.123301 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205"} err="failed to get container status \"7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205\": rpc error: code = NotFound desc = could not find container \"7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205\": container with ID starting with 7e7162eae02e7b5faff9e3702639288f680941766aa76817063de069046de205 not found: ID does not exist" Sep 06 14:14:31.132157 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.132123 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595"] Sep 06 14:14:31.135591 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.135556 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dfd6b-8ddc7c6c4-2f595"] Sep 06 14:14:31.196669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.196632 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.196669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.196664 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xvl\" (UniqueName: \"kubernetes.io/projected/9045e982-5e27-4dd8-abfa-73fbce19c750-kube-api-access-m8xvl\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.196669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.196677 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045e982-5e27-4dd8-abfa-73fbce19c750-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.196964 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.196690 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9045e982-5e27-4dd8-abfa-73fbce19c750-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:31.236783 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:31.236747 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9045e982-5e27-4dd8-abfa-73fbce19c750" path="/var/lib/kubelet/pods/9045e982-5e27-4dd8-abfa-73fbce19c750/volumes" Sep 06 14:14:42.818824 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.818785 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp"] Sep 06 14:14:42.819500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.819247 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75417b8-c211-4b0d-b7d1-a21e9151e317" containerName="maas-api" Sep 06 14:14:42.819500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.819262 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75417b8-c211-4b0d-b7d1-a21e9151e317" containerName="maas-api" Sep 06 14:14:42.819500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.819275 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9045e982-5e27-4dd8-abfa-73fbce19c750" containerName="maas-api" Sep 06 14:14:42.819500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.819281 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9045e982-5e27-4dd8-abfa-73fbce19c750" containerName="maas-api" Sep 06 14:14:42.819500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.819370 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9045e982-5e27-4dd8-abfa-73fbce19c750" containerName="maas-api" Sep 06 14:14:42.819500 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.819383 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75417b8-c211-4b0d-b7d1-a21e9151e317" containerName="maas-api" Sep 06 14:14:42.824170 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.824141 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:42.826277 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.826249 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-81fae671-serving-cert\"" Sep 06 14:14:42.831624 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:42.831590 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp"] Sep 06 14:14:43.001590 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.001545 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.001590 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.001599 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759b700-4f12-41af-b0de-b3f7f81d5454-openshift-service-ca\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.001827 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.001615 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td47n\" (UniqueName: \"kubernetes.io/projected/2759b700-4f12-41af-b0de-b3f7f81d5454-kube-api-access-td47n\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.001827 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.001760 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-metrics-certs\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.103288 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.103192 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-metrics-certs\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.103288 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.103256 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.103288 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.103282 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759b700-4f12-41af-b0de-b3f7f81d5454-openshift-service-ca\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.103596 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.103300 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td47n\" (UniqueName: \"kubernetes.io/projected/2759b700-4f12-41af-b0de-b3f7f81d5454-kube-api-access-td47n\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.103596 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:43.103425 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-serving-cert: secret "maas-api-e2e-mt-81fae671-serving-cert" not found Sep 06 14:14:43.103596 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:43.103503 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls podName:2759b700-4f12-41af-b0de-b3f7f81d5454 nodeName:}" failed. No retries permitted until 2026-09-06 14:14:43.603481473 +0000 UTC m=+1032.866926184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls") pod "maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" (UID: "2759b700-4f12-41af-b0de-b3f7f81d5454") : secret "maas-api-e2e-mt-81fae671-serving-cert" not found Sep 06 14:14:43.103991 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.103974 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759b700-4f12-41af-b0de-b3f7f81d5454-openshift-service-ca\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.105947 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.105922 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-metrics-certs\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.114763 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.114737 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td47n\" (UniqueName: \"kubernetes.io/projected/2759b700-4f12-41af-b0de-b3f7f81d5454-kube-api-access-td47n\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.609313 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.609280 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.611730 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.611700 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls\") pod \"maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:43.738106 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:43.738067 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:44.015101 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:44.015075 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp"] Sep 06 14:14:44.016290 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:14:44.016255 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2759b700_4f12_41af_b0de_b3f7f81d5454.slice/crio-338b593617e7e82b078242b532570601feeca802735717bb7490699ff7bc14fc WatchSource:0}: Error finding container 338b593617e7e82b078242b532570601feeca802735717bb7490699ff7bc14fc: Status 404 returned error can't find the container with id 338b593617e7e82b078242b532570601feeca802735717bb7490699ff7bc14fc Sep 06 14:14:44.179999 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:44.179960 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" event={"ID":"2759b700-4f12-41af-b0de-b3f7f81d5454","Type":"ContainerStarted","Data":"338b593617e7e82b078242b532570601feeca802735717bb7490699ff7bc14fc"} Sep 06 14:14:45.187662 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:45.187623 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" event={"ID":"2759b700-4f12-41af-b0de-b3f7f81d5454","Type":"ContainerStarted","Data":"27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041"} Sep 06 14:14:45.188121 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:45.187756 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:45.210682 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:45.205325 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" podStartSLOduration=2.762786614 podStartE2EDuration="3.205299668s" totalImagesPullingTime="442.513054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:45.201936849 +0000 UTC m=+1034.465381573" watchObservedRunningTime="2026-09-06 14:14:45.205299668 +0000 UTC m=+1034.468744393" Sep 06 14:14:47.900136 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:47.900095 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8"] Sep 06 14:14:47.900693 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:47.900348 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" podUID="79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" containerName="maas-api" containerID="cri-o://fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a" gracePeriod=30 Sep 06 14:14:47.949800 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:47.949759 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc"] Sep 06 14:14:47.950038 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:47.950007 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" podUID="c3713bde-3e66-4a70-8a20-c2d941c06548" containerName="payload-processing" containerID="cri-o://7c0855a9345eca207350b57516f45ac80fb7762b647b9c5d62892ab4e3b6e46b" gracePeriod=30 Sep 06 14:14:48.208583 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.208550 2550 generic.go:356] "Generic (PLEG): container finished" podID="c3713bde-3e66-4a70-8a20-c2d941c06548" containerID="7c0855a9345eca207350b57516f45ac80fb7762b647b9c5d62892ab4e3b6e46b" exitCode=0 Sep 06 14:14:48.208730 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.208626 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" event={"ID":"c3713bde-3e66-4a70-8a20-c2d941c06548","Type":"ContainerDied","Data":"7c0855a9345eca207350b57516f45ac80fb7762b647b9c5d62892ab4e3b6e46b"} Sep 06 14:14:48.258842 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.258812 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:14:48.376017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.375985 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:14:48.457267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.457160 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmff4\" (UniqueName: \"kubernetes.io/projected/c3713bde-3e66-4a70-8a20-c2d941c06548-kube-api-access-hmff4\") pod \"c3713bde-3e66-4a70-8a20-c2d941c06548\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " Sep 06 14:14:48.457267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.457263 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3713bde-3e66-4a70-8a20-c2d941c06548-plugins-config-volume\") pod \"c3713bde-3e66-4a70-8a20-c2d941c06548\" (UID: \"c3713bde-3e66-4a70-8a20-c2d941c06548\") " Sep 06 14:14:48.457520 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.457362 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-872cd\" (UniqueName: \"kubernetes.io/projected/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-kube-api-access-872cd\") pod \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " Sep 06 14:14:48.457584 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.457450 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls\") pod \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " Sep 06 14:14:48.457713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.457687 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3713bde-3e66-4a70-8a20-c2d941c06548-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3713bde-3e66-4a70-8a20-c2d941c06548" (UID: "c3713bde-3e66-4a70-8a20-c2d941c06548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:48.457951 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.457926 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3713bde-3e66-4a70-8a20-c2d941c06548-plugins-config-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.459459 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.459430 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3713bde-3e66-4a70-8a20-c2d941c06548-kube-api-access-hmff4" (OuterVolumeSpecName: "kube-api-access-hmff4") pod "c3713bde-3e66-4a70-8a20-c2d941c06548" (UID: "c3713bde-3e66-4a70-8a20-c2d941c06548"). InnerVolumeSpecName "kube-api-access-hmff4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:48.459560 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.459534 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-kube-api-access-872cd" (OuterVolumeSpecName: "kube-api-access-872cd") pod "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" (UID: "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55"). InnerVolumeSpecName "kube-api-access-872cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:14:48.459618 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.459576 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" (UID: "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:14:48.558594 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.558555 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-metrics-certs\") pod \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " Sep 06 14:14:48.558772 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.558620 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-openshift-service-ca\") pod \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\" (UID: \"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55\") " Sep 06 14:14:48.558931 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.558879 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872cd\" (UniqueName: \"kubernetes.io/projected/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-kube-api-access-872cd\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.558982 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.558937 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.558982 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.558952 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmff4\" (UniqueName: \"kubernetes.io/projected/c3713bde-3e66-4a70-8a20-c2d941c06548-kube-api-access-hmff4\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.559054 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.558987 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" (UID: "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:14:48.560766 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.560739 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" (UID: "79a7bfb9-a7cb-467b-aaf6-165d26cb3a55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:14:48.659482 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.659436 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:48.659482 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:48.659474 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:14:49.216567 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.216532 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" Sep 06 14:14:49.216567 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.216546 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc" event={"ID":"c3713bde-3e66-4a70-8a20-c2d941c06548","Type":"ContainerDied","Data":"ffba62dda20de30b889e16b42f4952c2ffe3ec7559dc2ec36454c885639fe65b"} Sep 06 14:14:49.217126 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.216636 2550 scope.go:117] "RemoveContainer" containerID="7c0855a9345eca207350b57516f45ac80fb7762b647b9c5d62892ab4e3b6e46b" Sep 06 14:14:49.218161 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.218134 2550 generic.go:356] "Generic (PLEG): container finished" podID="79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" containerID="fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a" exitCode=0 Sep 06 14:14:49.218261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.218169 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" event={"ID":"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55","Type":"ContainerDied","Data":"fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a"} Sep 06 14:14:49.218261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.218194 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" Sep 06 14:14:49.218261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.218226 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8" event={"ID":"79a7bfb9-a7cb-467b-aaf6-165d26cb3a55","Type":"ContainerDied","Data":"2d45647bccceeaf1170db109afa4aef62d78124bf5631c53baa1d9e39efde22c"} Sep 06 14:14:49.226419 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.226399 2550 scope.go:117] "RemoveContainer" containerID="fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a" Sep 06 14:14:49.236759 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.236704 2550 scope.go:117] "RemoveContainer" containerID="fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a" Sep 06 14:14:49.237060 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:49.237033 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a\": container with ID starting with fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a not found: ID does not exist" containerID="fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a" Sep 06 14:14:49.237189 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.237076 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a"} err="failed to get container status \"fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a\": rpc error: code = NotFound desc = could not find container \"fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a\": container with ID starting with fb88a9966d46fef1eddd7cc17cfeacfdf5e29791465beabdb3ded3f39782073a not found: ID does not exist" Sep 06 14:14:49.242179 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.242152 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8"] Sep 06 14:14:49.245003 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.244977 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ec585-85d889bc8f-t4xl8"] Sep 06 14:14:49.253305 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.253268 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc"] Sep 06 14:14:49.255949 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:49.255924 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ec585-77789bc79c-wnkgc"] Sep 06 14:14:51.200199 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:51.200171 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:14:51.239623 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:51.239579 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" path="/var/lib/kubelet/pods/79a7bfb9-a7cb-467b-aaf6-165d26cb3a55/volumes" Sep 06 14:14:51.240188 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:51.240166 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3713bde-3e66-4a70-8a20-c2d941c06548" path="/var/lib/kubelet/pods/c3713bde-3e66-4a70-8a20-c2d941c06548/volumes" Sep 06 14:14:59.003179 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003142 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn"] Sep 06 14:14:59.003627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003579 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3713bde-3e66-4a70-8a20-c2d941c06548" containerName="payload-processing" Sep 06 14:14:59.003627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003591 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3713bde-3e66-4a70-8a20-c2d941c06548" containerName="payload-processing" Sep 06 14:14:59.003627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003606 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" containerName="maas-api" Sep 06 14:14:59.003627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003612 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" containerName="maas-api" Sep 06 14:14:59.003760 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003672 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3713bde-3e66-4a70-8a20-c2d941c06548" containerName="payload-processing" Sep 06 14:14:59.003760 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.003683 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a7bfb9-a7cb-467b-aaf6-165d26cb3a55" containerName="maas-api" Sep 06 14:14:59.006622 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.006605 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.008920 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.008871 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0aaff3-serving-cert\"" Sep 06 14:14:59.017462 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.017430 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn"] Sep 06 14:14:59.051002 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.050967 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pwn8\" (UniqueName: \"kubernetes.io/projected/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-kube-api-access-2pwn8\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.051208 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.051038 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.051208 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.051182 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.051285 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.051226 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.152691 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.152654 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.152922 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.152699 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.152922 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.152741 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwn8\" (UniqueName: \"kubernetes.io/projected/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-kube-api-access-2pwn8\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.152922 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.152798 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.152922 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:59.152826 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-serving-cert: secret "maas-api-e2e-shared-a-w3-0aaff3-serving-cert" not found Sep 06 14:14:59.152922 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:14:59.152920 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls podName:63fe9f6d-2c01-44bb-9ba7-32f2ca10603c nodeName:}" failed. No retries permitted until 2026-09-06 14:14:59.652882919 +0000 UTC m=+1048.916327630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls") pod "maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" (UID: "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c") : secret "maas-api-e2e-shared-a-w3-0aaff3-serving-cert" not found Sep 06 14:14:59.153541 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.153518 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.155346 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.155323 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.163148 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.163121 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwn8\" (UniqueName: \"kubernetes.io/projected/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-kube-api-access-2pwn8\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.657606 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.657562 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.659942 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.659917 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.763772 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.763735 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q"] Sep 06 14:14:59.768337 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.768316 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:14:59.770509 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.770482 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0aaff3-dockercfg-tw8lw\"" Sep 06 14:14:59.770647 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.770484 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0aaff3\"" Sep 06 14:14:59.775544 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.775515 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q"] Sep 06 14:14:59.860169 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.860128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7l4\" (UniqueName: \"kubernetes.io/projected/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-kube-api-access-9g7l4\") pod \"payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:14:59.860325 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.860232 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:14:59.918806 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.918773 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:14:59.961380 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.961336 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:14:59.961541 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.961455 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7l4\" (UniqueName: \"kubernetes.io/projected/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-kube-api-access-9g7l4\") pod \"payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:14:59.962112 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.962090 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:14:59.972089 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:14:59.972056 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7l4\" (UniqueName: \"kubernetes.io/projected/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-kube-api-access-9g7l4\") pod \"payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:15:00.083261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:00.083224 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:15:00.200416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:00.200385 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn"] Sep 06 14:15:00.202049 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:15:00.202020 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fe9f6d_2c01_44bb_9ba7_32f2ca10603c.slice/crio-29def9ba3eaeea2f9f691157640378cb5d7a85ee77afdb335eeecad31069f0bb WatchSource:0}: Error finding container 29def9ba3eaeea2f9f691157640378cb5d7a85ee77afdb335eeecad31069f0bb: Status 404 returned error can't find the container with id 29def9ba3eaeea2f9f691157640378cb5d7a85ee77afdb335eeecad31069f0bb Sep 06 14:15:00.292621 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:00.292583 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" event={"ID":"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c","Type":"ContainerStarted","Data":"29def9ba3eaeea2f9f691157640378cb5d7a85ee77afdb335eeecad31069f0bb"} Sep 06 14:15:00.358644 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:00.358619 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q"] Sep 06 14:15:00.360214 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:15:00.360182 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff39c708_38d5_4084_aa9f_dfa8d3241cdd.slice/crio-34994114026f8190a777a468b424fca79fbd04331620cb6c0e8eb38236889353 WatchSource:0}: Error finding container 34994114026f8190a777a468b424fca79fbd04331620cb6c0e8eb38236889353: Status 404 returned error can't find the container with id 34994114026f8190a777a468b424fca79fbd04331620cb6c0e8eb38236889353 Sep 06 14:15:01.300627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.300590 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" event={"ID":"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c","Type":"ContainerStarted","Data":"4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58"} Sep 06 14:15:01.301130 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.300710 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:15:01.302296 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.302271 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" event={"ID":"ff39c708-38d5-4084-aa9f-dfa8d3241cdd","Type":"ContainerStarted","Data":"6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab"} Sep 06 14:15:01.302401 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.302312 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" event={"ID":"ff39c708-38d5-4084-aa9f-dfa8d3241cdd","Type":"ContainerStarted","Data":"34994114026f8190a777a468b424fca79fbd04331620cb6c0e8eb38236889353"} Sep 06 14:15:01.302401 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.302363 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:15:01.316227 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.316170 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" podStartSLOduration=2.833472231 podStartE2EDuration="3.316150343s" totalImagesPullingTime="482.678112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:01.31440162 +0000 UTC m=+1050.577846344" watchObservedRunningTime="2026-09-06 14:15:01.316150343 +0000 UTC m=+1050.579595067" Sep 06 14:15:01.330782 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:01.330715 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" podStartSLOduration=2.330690967 podStartE2EDuration="2.330690967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:01.328334458 +0000 UTC m=+1050.591779183" watchObservedRunningTime="2026-09-06 14:15:01.330690967 +0000 UTC m=+1050.594135691" Sep 06 14:15:07.314939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:07.314874 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:15:12.312045 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:12.312014 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:15:24.378201 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.378108 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp"] Sep 06 14:15:24.378815 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.378357 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" podUID="2759b700-4f12-41af-b0de-b3f7f81d5454" containerName="maas-api" containerID="cri-o://27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041" gracePeriod=30 Sep 06 14:15:24.653436 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.653356 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:15:24.806145 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.806108 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td47n\" (UniqueName: \"kubernetes.io/projected/2759b700-4f12-41af-b0de-b3f7f81d5454-kube-api-access-td47n\") pod \"2759b700-4f12-41af-b0de-b3f7f81d5454\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " Sep 06 14:15:24.806350 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.806190 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759b700-4f12-41af-b0de-b3f7f81d5454-openshift-service-ca\") pod \"2759b700-4f12-41af-b0de-b3f7f81d5454\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " Sep 06 14:15:24.806350 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.806289 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls\") pod \"2759b700-4f12-41af-b0de-b3f7f81d5454\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " Sep 06 14:15:24.806350 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.806346 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-metrics-certs\") pod \"2759b700-4f12-41af-b0de-b3f7f81d5454\" (UID: \"2759b700-4f12-41af-b0de-b3f7f81d5454\") " Sep 06 14:15:24.806603 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.806577 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2759b700-4f12-41af-b0de-b3f7f81d5454-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2759b700-4f12-41af-b0de-b3f7f81d5454" (UID: "2759b700-4f12-41af-b0de-b3f7f81d5454"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:15:24.808574 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.808540 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2759b700-4f12-41af-b0de-b3f7f81d5454-kube-api-access-td47n" (OuterVolumeSpecName: "kube-api-access-td47n") pod "2759b700-4f12-41af-b0de-b3f7f81d5454" (UID: "2759b700-4f12-41af-b0de-b3f7f81d5454"). InnerVolumeSpecName "kube-api-access-td47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:15:24.808574 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.808547 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2759b700-4f12-41af-b0de-b3f7f81d5454" (UID: "2759b700-4f12-41af-b0de-b3f7f81d5454"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:15:24.808574 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.808569 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2759b700-4f12-41af-b0de-b3f7f81d5454" (UID: "2759b700-4f12-41af-b0de-b3f7f81d5454"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:15:24.907600 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.907496 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td47n\" (UniqueName: \"kubernetes.io/projected/2759b700-4f12-41af-b0de-b3f7f81d5454-kube-api-access-td47n\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.907600 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.907530 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759b700-4f12-41af-b0de-b3f7f81d5454-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.907600 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.907546 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:15:24.907600 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:24.907558 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759b700-4f12-41af-b0de-b3f7f81d5454-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:15:25.458038 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.458006 2550 generic.go:356] "Generic (PLEG): container finished" podID="2759b700-4f12-41af-b0de-b3f7f81d5454" containerID="27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041" exitCode=0 Sep 06 14:15:25.458548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.458068 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" Sep 06 14:15:25.458548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.458098 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" event={"ID":"2759b700-4f12-41af-b0de-b3f7f81d5454","Type":"ContainerDied","Data":"27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041"} Sep 06 14:15:25.458548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.458154 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp" event={"ID":"2759b700-4f12-41af-b0de-b3f7f81d5454","Type":"ContainerDied","Data":"338b593617e7e82b078242b532570601feeca802735717bb7490699ff7bc14fc"} Sep 06 14:15:25.458548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.458176 2550 scope.go:117] "RemoveContainer" containerID="27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041" Sep 06 14:15:25.468549 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.468521 2550 scope.go:117] "RemoveContainer" containerID="27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041" Sep 06 14:15:25.468935 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:15:25.468915 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041\": container with ID starting with 27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041 not found: ID does not exist" containerID="27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041" Sep 06 14:15:25.469001 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.468944 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041"} err="failed to get container status \"27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041\": rpc error: code = NotFound desc = could not find container \"27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041\": container with ID starting with 27cab280c8e363fc944200be6541877d33080ad590aea9c9422b87a08a969041 not found: ID does not exist" Sep 06 14:15:25.473868 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.473838 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp"] Sep 06 14:15:25.479445 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:25.479391 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81fae671-7fcb779cfd-j68kp"] Sep 06 14:15:27.102103 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.102069 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz"] Sep 06 14:15:27.102549 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.102535 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2759b700-4f12-41af-b0de-b3f7f81d5454" containerName="maas-api" Sep 06 14:15:27.102549 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.102551 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2759b700-4f12-41af-b0de-b3f7f81d5454" containerName="maas-api" Sep 06 14:15:27.102625 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.102610 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2759b700-4f12-41af-b0de-b3f7f81d5454" containerName="maas-api" Sep 06 14:15:27.107234 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.107210 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.109242 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.109222 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c443d8-serving-cert\"" Sep 06 14:15:27.118598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.118573 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz"] Sep 06 14:15:27.232064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.232025 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11127b7-5959-444e-af44-247ad72bd39a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.232064 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.232062 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgbn\" (UniqueName: \"kubernetes.io/projected/a11127b7-5959-444e-af44-247ad72bd39a-kube-api-access-vlgbn\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.232324 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.232099 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.232324 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.232141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.237007 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.236975 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2759b700-4f12-41af-b0de-b3f7f81d5454" path="/var/lib/kubelet/pods/2759b700-4f12-41af-b0de-b3f7f81d5454/volumes" Sep 06 14:15:27.333662 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.333624 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.333855 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.333736 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11127b7-5959-444e-af44-247ad72bd39a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.333855 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.333767 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgbn\" (UniqueName: \"kubernetes.io/projected/a11127b7-5959-444e-af44-247ad72bd39a-kube-api-access-vlgbn\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.333855 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:15:27.333783 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-serving-cert: secret "maas-api-e2e-shared-b-w3-c443d8-serving-cert" not found Sep 06 14:15:27.333855 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.333824 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.333855 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:15:27.333850 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls podName:a11127b7-5959-444e-af44-247ad72bd39a nodeName:}" failed. No retries permitted until 2026-09-06 14:15:27.833828478 +0000 UTC m=+1077.097273196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" (UID: "a11127b7-5959-444e-af44-247ad72bd39a") : secret "maas-api-e2e-shared-b-w3-c443d8-serving-cert" not found Sep 06 14:15:27.334472 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.334447 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11127b7-5959-444e-af44-247ad72bd39a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.336376 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.336356 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.343229 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.343193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgbn\" (UniqueName: \"kubernetes.io/projected/a11127b7-5959-444e-af44-247ad72bd39a-kube-api-access-vlgbn\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.839223 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.839179 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:27.842532 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:27.842494 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:28.020195 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:28.020154 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:28.302078 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:28.302048 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz"] Sep 06 14:15:28.303856 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:15:28.303828 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11127b7_5959_444e_af44_247ad72bd39a.slice/crio-1c73d322de49a7024e9a5e3cf79d06ae24856b5f79bcfc8150748fe3ccba4897 WatchSource:0}: Error finding container 1c73d322de49a7024e9a5e3cf79d06ae24856b5f79bcfc8150748fe3ccba4897: Status 404 returned error can't find the container with id 1c73d322de49a7024e9a5e3cf79d06ae24856b5f79bcfc8150748fe3ccba4897 Sep 06 14:15:28.477772 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:28.477730 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" event={"ID":"a11127b7-5959-444e-af44-247ad72bd39a","Type":"ContainerStarted","Data":"1c73d322de49a7024e9a5e3cf79d06ae24856b5f79bcfc8150748fe3ccba4897"} Sep 06 14:15:29.486220 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:29.486175 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" event={"ID":"a11127b7-5959-444e-af44-247ad72bd39a","Type":"ContainerStarted","Data":"0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906"} Sep 06 14:15:29.486668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:29.486341 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:15:29.503280 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:29.503223 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" podStartSLOduration=2.198888772 podStartE2EDuration="2.503203642s" totalImagesPullingTime="304.31487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:29.500173079 +0000 UTC m=+1078.763617803" watchObservedRunningTime="2026-09-06 14:15:29.503203642 +0000 UTC m=+1078.766648366" Sep 06 14:15:35.498998 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:15:35.498970 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:16:16.387586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.387545 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6"] Sep 06 14:16:16.392320 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.392296 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.395081 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.394545 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a388e1-serving-cert\"" Sep 06 14:16:16.402996 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.402970 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6"] Sep 06 14:16:16.411706 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.411661 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24344bb6-dccb-442a-9212-2e7a25b38654-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.412037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.411826 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.412037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.411867 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.412037 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.411961 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z792m\" (UniqueName: \"kubernetes.io/projected/24344bb6-dccb-442a-9212-2e7a25b38654-kube-api-access-z792m\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.513052 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.513012 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24344bb6-dccb-442a-9212-2e7a25b38654-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.513238 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.513086 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.513238 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.513115 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.513238 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.513174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z792m\" (UniqueName: \"kubernetes.io/projected/24344bb6-dccb-442a-9212-2e7a25b38654-kube-api-access-z792m\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.513634 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:16:16.513605 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-serving-cert: secret "maas-api-e2e-shared-b-w2-a388e1-serving-cert" not found Sep 06 14:16:16.513884 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:16:16.513868 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls podName:24344bb6-dccb-442a-9212-2e7a25b38654 nodeName:}" failed. No retries permitted until 2026-09-06 14:16:17.013835332 +0000 UTC m=+1126.277280038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" (UID: "24344bb6-dccb-442a-9212-2e7a25b38654") : secret "maas-api-e2e-shared-b-w2-a388e1-serving-cert" not found Sep 06 14:16:16.514140 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.514112 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24344bb6-dccb-442a-9212-2e7a25b38654-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.522072 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.516842 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:16.523053 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:16.523021 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z792m\" (UniqueName: \"kubernetes.io/projected/24344bb6-dccb-442a-9212-2e7a25b38654-kube-api-access-z792m\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:17.018396 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.018357 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:17.020852 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.020826 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:17.103920 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.103870 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96"] Sep 06 14:16:17.108017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.107988 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.110260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.110235 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a388e1-dockercfg-wvzjc\"" Sep 06 14:16:17.110391 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.110246 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a388e1\"" Sep 06 14:16:17.116362 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.116333 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96"] Sep 06 14:16:17.219687 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.219646 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nh8h\" (UniqueName: \"kubernetes.io/projected/b2d704db-29e0-45fa-91ad-98b2ca05ba85-kube-api-access-5nh8h\") pod \"payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.219864 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.219712 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d704db-29e0-45fa-91ad-98b2ca05ba85-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.307422 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.307321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:17.321012 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.320965 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nh8h\" (UniqueName: \"kubernetes.io/projected/b2d704db-29e0-45fa-91ad-98b2ca05ba85-kube-api-access-5nh8h\") pod \"payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.321213 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.321188 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d704db-29e0-45fa-91ad-98b2ca05ba85-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.322189 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.322152 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d704db-29e0-45fa-91ad-98b2ca05ba85-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.328629 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.328602 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nh8h\" (UniqueName: \"kubernetes.io/projected/b2d704db-29e0-45fa-91ad-98b2ca05ba85-kube-api-access-5nh8h\") pod \"payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.421038 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.421000 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.586679 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.586649 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6"] Sep 06 14:16:17.588162 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:16:17.588135 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24344bb6_dccb_442a_9212_2e7a25b38654.slice/crio-66c351333004ea384b4ceb53f80be9f658859d4af967c011e3ee375aa465855e WatchSource:0}: Error finding container 66c351333004ea384b4ceb53f80be9f658859d4af967c011e3ee375aa465855e: Status 404 returned error can't find the container with id 66c351333004ea384b4ceb53f80be9f658859d4af967c011e3ee375aa465855e Sep 06 14:16:17.700813 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.700776 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96"] Sep 06 14:16:17.702472 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:16:17.702443 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d704db_29e0_45fa_91ad_98b2ca05ba85.slice/crio-a4f3b73b2897f5d42a8a12a2d4c54e3f604c7a3fded68878cbbc96cf474e8d8f WatchSource:0}: Error finding container a4f3b73b2897f5d42a8a12a2d4c54e3f604c7a3fded68878cbbc96cf474e8d8f: Status 404 returned error can't find the container with id a4f3b73b2897f5d42a8a12a2d4c54e3f604c7a3fded68878cbbc96cf474e8d8f Sep 06 14:16:17.798012 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.797971 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" event={"ID":"b2d704db-29e0-45fa-91ad-98b2ca05ba85","Type":"ContainerStarted","Data":"63274143fe007b232e87ed7f4985f1dfadefc7b4da4033bba9829c27bfb8e113"} Sep 06 14:16:17.798191 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.798043 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" event={"ID":"b2d704db-29e0-45fa-91ad-98b2ca05ba85","Type":"ContainerStarted","Data":"a4f3b73b2897f5d42a8a12a2d4c54e3f604c7a3fded68878cbbc96cf474e8d8f"} Sep 06 14:16:17.798191 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.798171 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:17.799484 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.799452 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" event={"ID":"24344bb6-dccb-442a-9212-2e7a25b38654","Type":"ContainerStarted","Data":"66c351333004ea384b4ceb53f80be9f658859d4af967c011e3ee375aa465855e"} Sep 06 14:16:17.813421 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:17.813365 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" podStartSLOduration=0.813345014 podStartE2EDuration="813.345014ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:16:17.812491875 +0000 UTC m=+1127.075936601" watchObservedRunningTime="2026-09-06 14:16:17.813345014 +0000 UTC m=+1127.076789738" Sep 06 14:16:18.807136 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:18.807100 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" event={"ID":"24344bb6-dccb-442a-9212-2e7a25b38654","Type":"ContainerStarted","Data":"869ea0b8ca3c7cd8b5ec15b9f4b7f5d171b6bc1e88c12bba7a6448bc3229d252"} Sep 06 14:16:18.825953 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:18.825880 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" podStartSLOduration=2.505044672 podStartE2EDuration="2.825864362s" totalImagesPullingTime="320.81969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:16:18.825006956 +0000 UTC m=+1128.088451682" watchObservedRunningTime="2026-09-06 14:16:18.825864362 +0000 UTC m=+1128.089309085" Sep 06 14:16:19.813575 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:19.813542 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:25.826463 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:25.826434 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:16:28.809394 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:28.809363 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:16:31.405616 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:31.405585 2550 scope.go:117] "RemoveContainer" containerID="bb0aecae8fab570cd8c2d1b2f90a1c402cfec2ac5193a1b355bfed3b1e255365" Sep 06 14:16:46.872129 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:46.872086 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7542-140355587229712"] Sep 06 14:16:46.877514 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:46.877400 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:46.879569 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:46.879542 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\"" Sep 06 14:16:46.880977 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:46.880945 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"] Sep 06 14:16:47.005691 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:47.005649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbhx\" (UniqueName: \"kubernetes.io/projected/1728300e-3827-4830-99d6-f092b2410e7c-kube-api-access-lbbhx\") pod \"test-curl-7542-140355587229712\" (UID: \"1728300e-3827-4830-99d6-f092b2410e7c\") " pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:47.106536 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:47.106491 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbhx\" (UniqueName: \"kubernetes.io/projected/1728300e-3827-4830-99d6-f092b2410e7c-kube-api-access-lbbhx\") pod \"test-curl-7542-140355587229712\" (UID: \"1728300e-3827-4830-99d6-f092b2410e7c\") " pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:47.113970 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:47.113936 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbhx\" (UniqueName: \"kubernetes.io/projected/1728300e-3827-4830-99d6-f092b2410e7c-kube-api-access-lbbhx\") pod \"test-curl-7542-140355587229712\" (UID: \"1728300e-3827-4830-99d6-f092b2410e7c\") " pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:47.188959 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:47.188919 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:47.470303 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:47.470276 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"] Sep 06 14:16:47.471533 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:16:47.471506 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1728300e_3827_4830_99d6_f092b2410e7c.slice/crio-ba8950311f297986e5a77ec3d1d42a365ad5a70213c64bdd73adfa8e13c42af7 WatchSource:0}: Error finding container ba8950311f297986e5a77ec3d1d42a365ad5a70213c64bdd73adfa8e13c42af7: Status 404 returned error can't find the container with id ba8950311f297986e5a77ec3d1d42a365ad5a70213c64bdd73adfa8e13c42af7 Sep 06 14:16:47.991039 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:47.991006 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229712" event={"ID":"1728300e-3827-4830-99d6-f092b2410e7c","Type":"ContainerStarted","Data":"ba8950311f297986e5a77ec3d1d42a365ad5a70213c64bdd73adfa8e13c42af7"} Sep 06 14:16:48.999758 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:48.999722 2550 generic.go:356] "Generic (PLEG): container finished" podID="1728300e-3827-4830-99d6-f092b2410e7c" containerID="d064fc1a0bb871b43b8ee1c2ebe1b68c0357e55100dbe23ac5bf16cfada0a935" exitCode=0 Sep 06 14:16:49.000162 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:48.999812 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229712" event={"ID":"1728300e-3827-4830-99d6-f092b2410e7c","Type":"ContainerDied","Data":"d064fc1a0bb871b43b8ee1c2ebe1b68c0357e55100dbe23ac5bf16cfada0a935"} Sep 06 14:16:50.149368 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.149328 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:50.181306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.181277 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7542-140355587229712_1728300e-3827-4830-99d6-f092b2410e7c/test-curl-7542-140355587229712/0.log" Sep 06 14:16:50.203950 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.203917 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"] Sep 06 14:16:50.206422 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.206400 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"] Sep 06 14:16:50.234424 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.234387 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbbhx\" (UniqueName: \"kubernetes.io/projected/1728300e-3827-4830-99d6-f092b2410e7c-kube-api-access-lbbhx\") pod \"1728300e-3827-4830-99d6-f092b2410e7c\" (UID: \"1728300e-3827-4830-99d6-f092b2410e7c\") " Sep 06 14:16:50.236565 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.236529 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1728300e-3827-4830-99d6-f092b2410e7c-kube-api-access-lbbhx" (OuterVolumeSpecName: "kube-api-access-lbbhx") pod "1728300e-3827-4830-99d6-f092b2410e7c" (UID: "1728300e-3827-4830-99d6-f092b2410e7c"). InnerVolumeSpecName "kube-api-access-lbbhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:16:50.335691 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:50.335594 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbhx\" (UniqueName: \"kubernetes.io/projected/1728300e-3827-4830-99d6-f092b2410e7c-kube-api-access-lbbhx\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:16:51.013519 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:51.013488 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:51.013519 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:51.013505 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8950311f297986e5a77ec3d1d42a365ad5a70213c64bdd73adfa8e13c42af7" Sep 06 14:16:51.027336 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:16:51.027303 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7542-140355587229712\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="1728300e-3827-4830-99d6-f092b2410e7c" pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:51.235403 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:16:51.235367 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7542-140355587229712\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="1728300e-3827-4830-99d6-f092b2410e7c" pod="opendatahub/test-curl-7542-140355587229712" Sep 06 14:16:51.237349 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:51.237319 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1728300e-3827-4830-99d6-f092b2410e7c" path="/var/lib/kubelet/pods/1728300e-3827-4830-99d6-f092b2410e7c/volumes" Sep 06 14:16:59.790127 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.790036 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748"] Sep 06 14:16:59.790501 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.790479 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1728300e-3827-4830-99d6-f092b2410e7c" containerName="test-curl-7542-140355587229712" Sep 06 14:16:59.790501 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.790491 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1728300e-3827-4830-99d6-f092b2410e7c" containerName="test-curl-7542-140355587229712" Sep 06 14:16:59.790574 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.790565 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1728300e-3827-4830-99d6-f092b2410e7c" containerName="test-curl-7542-140355587229712" Sep 06 14:16:59.793609 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.793591 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:16:59.796272 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.796231 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25ae160a-dockercfg-2z8qb\"" Sep 06 14:16:59.796414 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.796314 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25ae160a\"" Sep 06 14:16:59.804260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.804228 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748"] Sep 06 14:16:59.828092 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.828053 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv88w\" (UniqueName: \"kubernetes.io/projected/6ac0f96d-ac7e-477b-9e24-f8272416ea29-kube-api-access-rv88w\") pod \"payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:16:59.828416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.828386 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac0f96d-ac7e-477b-9e24-f8272416ea29-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:16:59.831580 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.831555 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn"] Sep 06 14:16:59.835392 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.835360 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:16:59.845588 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.845552 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn"] Sep 06 14:16:59.929105 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.929070 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26b4\" (UniqueName: \"kubernetes.io/projected/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-kube-api-access-c26b4\") pod \"payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:16:59.929299 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.929118 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv88w\" (UniqueName: \"kubernetes.io/projected/6ac0f96d-ac7e-477b-9e24-f8272416ea29-kube-api-access-rv88w\") pod \"payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:16:59.929299 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.929179 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-plugins-config-volume\") pod \"payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:16:59.929299 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.929206 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac0f96d-ac7e-477b-9e24-f8272416ea29-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:16:59.930007 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.929986 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac0f96d-ac7e-477b-9e24-f8272416ea29-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:16:59.936158 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:16:59.936132 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv88w\" (UniqueName: \"kubernetes.io/projected/6ac0f96d-ac7e-477b-9e24-f8272416ea29-kube-api-access-rv88w\") pod \"payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:17:00.029902 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.029864 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26b4\" (UniqueName: \"kubernetes.io/projected/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-kube-api-access-c26b4\") pod \"payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:00.030125 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.029984 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-plugins-config-volume\") pod \"payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:00.030694 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.030673 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-plugins-config-volume\") pod \"payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:00.037642 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.037607 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26b4\" (UniqueName: \"kubernetes.io/projected/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-kube-api-access-c26b4\") pod \"payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:00.105863 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.105753 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:17:00.150531 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.150488 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:00.382738 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.382712 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748"] Sep 06 14:17:00.384612 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:17:00.384576 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac0f96d_ac7e_477b_9e24_f8272416ea29.slice/crio-140f73f4794ffe0bae499f0266ab8f6f24099753f05beb087e0d688c9db0d24f WatchSource:0}: Error finding container 140f73f4794ffe0bae499f0266ab8f6f24099753f05beb087e0d688c9db0d24f: Status 404 returned error can't find the container with id 140f73f4794ffe0bae499f0266ab8f6f24099753f05beb087e0d688c9db0d24f Sep 06 14:17:00.448227 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:00.448198 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn"] Sep 06 14:17:00.448480 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:17:00.448459 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f27a1e_dac5_40bb_a9a0_1360d2df0da2.slice/crio-fc5dd3bf4d95a2f8ab47908a43e13fe49d3f5694ef8ce246fb462a1bbf085383 WatchSource:0}: Error finding container fc5dd3bf4d95a2f8ab47908a43e13fe49d3f5694ef8ce246fb462a1bbf085383: Status 404 returned error can't find the container with id fc5dd3bf4d95a2f8ab47908a43e13fe49d3f5694ef8ce246fb462a1bbf085383 Sep 06 14:17:01.079651 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.079612 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" event={"ID":"68f27a1e-dac5-40bb-a9a0-1360d2df0da2","Type":"ContainerStarted","Data":"24c578e296808b7eda452b0535d16a650f8df1702f15945624b8d59ab7116a4b"} Sep 06 14:17:01.080155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.079670 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" event={"ID":"68f27a1e-dac5-40bb-a9a0-1360d2df0da2","Type":"ContainerStarted","Data":"fc5dd3bf4d95a2f8ab47908a43e13fe49d3f5694ef8ce246fb462a1bbf085383"} Sep 06 14:17:01.080155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.079803 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:01.081492 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.081463 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" event={"ID":"6ac0f96d-ac7e-477b-9e24-f8272416ea29","Type":"ContainerStarted","Data":"9b3d15ddec96224750af0af59c892f937795eaea8206a2977d78faea8a004d83"} Sep 06 14:17:01.081651 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.081510 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" event={"ID":"6ac0f96d-ac7e-477b-9e24-f8272416ea29","Type":"ContainerStarted","Data":"140f73f4794ffe0bae499f0266ab8f6f24099753f05beb087e0d688c9db0d24f"} Sep 06 14:17:01.081651 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.081549 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:17:01.095046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.094997 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" podStartSLOduration=2.094979177 podStartE2EDuration="2.094979177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:17:01.093651572 +0000 UTC m=+1170.357096299" watchObservedRunningTime="2026-09-06 14:17:01.094979177 +0000 UTC m=+1170.358423927" Sep 06 14:17:01.108084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:01.108032 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" podStartSLOduration=2.10801414 podStartE2EDuration="2.10801414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:17:01.106495204 +0000 UTC m=+1170.369939929" watchObservedRunningTime="2026-09-06 14:17:01.10801414 +0000 UTC m=+1170.371458863" Sep 06 14:17:12.089926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:12.089872 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:17:12.090350 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:12.090098 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:17:31.297642 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:31.297610 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:17:31.300827 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:17:31.300803 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:18:17.416277 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.416230 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn"] Sep 06 14:18:17.416782 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.416549 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" podUID="68f27a1e-dac5-40bb-a9a0-1360d2df0da2" containerName="payload-processing" containerID="cri-o://24c578e296808b7eda452b0535d16a650f8df1702f15945624b8d59ab7116a4b" gracePeriod=30 Sep 06 14:18:17.425481 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.424636 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748"] Sep 06 14:18:17.425481 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.424961 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" podUID="6ac0f96d-ac7e-477b-9e24-f8272416ea29" containerName="payload-pre-processing" containerID="cri-o://9b3d15ddec96224750af0af59c892f937795eaea8206a2977d78faea8a004d83" gracePeriod=30 Sep 06 14:18:17.588405 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.588362 2550 generic.go:356] "Generic (PLEG): container finished" podID="68f27a1e-dac5-40bb-a9a0-1360d2df0da2" containerID="24c578e296808b7eda452b0535d16a650f8df1702f15945624b8d59ab7116a4b" exitCode=0 Sep 06 14:18:17.588595 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.588463 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" event={"ID":"68f27a1e-dac5-40bb-a9a0-1360d2df0da2","Type":"ContainerDied","Data":"24c578e296808b7eda452b0535d16a650f8df1702f15945624b8d59ab7116a4b"} Sep 06 14:18:17.590352 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.590323 2550 generic.go:356] "Generic (PLEG): container finished" podID="6ac0f96d-ac7e-477b-9e24-f8272416ea29" containerID="9b3d15ddec96224750af0af59c892f937795eaea8206a2977d78faea8a004d83" exitCode=0 Sep 06 14:18:17.590505 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.590438 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" event={"ID":"6ac0f96d-ac7e-477b-9e24-f8272416ea29","Type":"ContainerDied","Data":"9b3d15ddec96224750af0af59c892f937795eaea8206a2977d78faea8a004d83"} Sep 06 14:18:17.703659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.703632 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:18:17.729641 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.729614 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:18:17.879655 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.879609 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv88w\" (UniqueName: \"kubernetes.io/projected/6ac0f96d-ac7e-477b-9e24-f8272416ea29-kube-api-access-rv88w\") pod \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " Sep 06 14:18:17.879835 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.879671 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c26b4\" (UniqueName: \"kubernetes.io/projected/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-kube-api-access-c26b4\") pod \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " Sep 06 14:18:17.879835 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.879744 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac0f96d-ac7e-477b-9e24-f8272416ea29-plugins-config-volume\") pod \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\" (UID: \"6ac0f96d-ac7e-477b-9e24-f8272416ea29\") " Sep 06 14:18:17.879835 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.879759 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-plugins-config-volume\") pod \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\" (UID: \"68f27a1e-dac5-40bb-a9a0-1360d2df0da2\") " Sep 06 14:18:17.880241 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.880203 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68f27a1e-dac5-40bb-a9a0-1360d2df0da2" (UID: "68f27a1e-dac5-40bb-a9a0-1360d2df0da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:18:17.880241 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.880209 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac0f96d-ac7e-477b-9e24-f8272416ea29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ac0f96d-ac7e-477b-9e24-f8272416ea29" (UID: "6ac0f96d-ac7e-477b-9e24-f8272416ea29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:18:17.881935 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.881905 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac0f96d-ac7e-477b-9e24-f8272416ea29-kube-api-access-rv88w" (OuterVolumeSpecName: "kube-api-access-rv88w") pod "6ac0f96d-ac7e-477b-9e24-f8272416ea29" (UID: "6ac0f96d-ac7e-477b-9e24-f8272416ea29"). InnerVolumeSpecName "kube-api-access-rv88w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:18:17.882043 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.881938 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-kube-api-access-c26b4" (OuterVolumeSpecName: "kube-api-access-c26b4") pod "68f27a1e-dac5-40bb-a9a0-1360d2df0da2" (UID: "68f27a1e-dac5-40bb-a9a0-1360d2df0da2"). InnerVolumeSpecName "kube-api-access-c26b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:18:17.981143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.981051 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv88w\" (UniqueName: \"kubernetes.io/projected/6ac0f96d-ac7e-477b-9e24-f8272416ea29-kube-api-access-rv88w\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:18:17.981143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.981084 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26b4\" (UniqueName: \"kubernetes.io/projected/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-kube-api-access-c26b4\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:18:17.981143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.981096 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac0f96d-ac7e-477b-9e24-f8272416ea29-plugins-config-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:18:17.981143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:17.981105 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f27a1e-dac5-40bb-a9a0-1360d2df0da2-plugins-config-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:18:18.598120 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.598081 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" Sep 06 14:18:18.598120 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.598090 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn" event={"ID":"68f27a1e-dac5-40bb-a9a0-1360d2df0da2","Type":"ContainerDied","Data":"fc5dd3bf4d95a2f8ab47908a43e13fe49d3f5694ef8ce246fb462a1bbf085383"} Sep 06 14:18:18.598662 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.598160 2550 scope.go:117] "RemoveContainer" containerID="24c578e296808b7eda452b0535d16a650f8df1702f15945624b8d59ab7116a4b" Sep 06 14:18:18.599616 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.599593 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" event={"ID":"6ac0f96d-ac7e-477b-9e24-f8272416ea29","Type":"ContainerDied","Data":"140f73f4794ffe0bae499f0266ab8f6f24099753f05beb087e0d688c9db0d24f"} Sep 06 14:18:18.599616 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.599613 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748" Sep 06 14:18:18.608182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.608160 2550 scope.go:117] "RemoveContainer" containerID="9b3d15ddec96224750af0af59c892f937795eaea8206a2977d78faea8a004d83" Sep 06 14:18:18.624433 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.624399 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748"] Sep 06 14:18:18.628456 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.628421 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25ae160a-845ddf58f4-fj748"] Sep 06 14:18:18.637522 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.637458 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn"] Sep 06 14:18:18.640205 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:18.640171 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25ae160a-9d658c4d5-ccqxn"] Sep 06 14:18:19.236805 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:19.236762 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f27a1e-dac5-40bb-a9a0-1360d2df0da2" path="/var/lib/kubelet/pods/68f27a1e-dac5-40bb-a9a0-1360d2df0da2/volumes" Sep 06 14:18:19.237369 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:19.237335 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac0f96d-ac7e-477b-9e24-f8272416ea29" path="/var/lib/kubelet/pods/6ac0f96d-ac7e-477b-9e24-f8272416ea29/volumes" Sep 06 14:18:35.542842 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.542796 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7545-cbd733"] Sep 06 14:18:35.543348 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.543330 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac0f96d-ac7e-477b-9e24-f8272416ea29" containerName="payload-pre-processing" Sep 06 14:18:35.543406 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.543350 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac0f96d-ac7e-477b-9e24-f8272416ea29" containerName="payload-pre-processing" Sep 06 14:18:35.543406 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.543385 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f27a1e-dac5-40bb-a9a0-1360d2df0da2" containerName="payload-processing" Sep 06 14:18:35.543406 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.543391 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f27a1e-dac5-40bb-a9a0-1360d2df0da2" containerName="payload-processing" Sep 06 14:18:35.543508 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.543467 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac0f96d-ac7e-477b-9e24-f8272416ea29" containerName="payload-pre-processing" Sep 06 14:18:35.543508 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.543474 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f27a1e-dac5-40bb-a9a0-1360d2df0da2" containerName="payload-processing" Sep 06 14:18:35.548249 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.548217 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:35.549820 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.549787 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7545-cbd733"] Sep 06 14:18:35.550841 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.550819 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\"" Sep 06 14:18:35.644250 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.644211 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7clmm\" (UniqueName: \"kubernetes.io/projected/e84e1e08-7415-4d15-82a2-4ba4097ea8c0-kube-api-access-7clmm\") pod \"mt-curl-7545-cbd733\" (UID: \"e84e1e08-7415-4d15-82a2-4ba4097ea8c0\") " pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:35.745154 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.745118 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7clmm\" (UniqueName: \"kubernetes.io/projected/e84e1e08-7415-4d15-82a2-4ba4097ea8c0-kube-api-access-7clmm\") pod \"mt-curl-7545-cbd733\" (UID: \"e84e1e08-7415-4d15-82a2-4ba4097ea8c0\") " pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:35.752598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.752569 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7clmm\" (UniqueName: \"kubernetes.io/projected/e84e1e08-7415-4d15-82a2-4ba4097ea8c0-kube-api-access-7clmm\") pod \"mt-curl-7545-cbd733\" (UID: \"e84e1e08-7415-4d15-82a2-4ba4097ea8c0\") " pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:35.860194 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:35.860097 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:36.136788 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:36.136756 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7545-cbd733"] Sep 06 14:18:36.138569 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:18:36.138540 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode84e1e08_7415_4d15_82a2_4ba4097ea8c0.slice/crio-7234edb454f8a3d051c054605e2e2441cee50a207f77dfcc6fa30cc635753bdb WatchSource:0}: Error finding container 7234edb454f8a3d051c054605e2e2441cee50a207f77dfcc6fa30cc635753bdb: Status 404 returned error can't find the container with id 7234edb454f8a3d051c054605e2e2441cee50a207f77dfcc6fa30cc635753bdb Sep 06 14:18:36.140156 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:36.140134 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:18:36.722708 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:36.722673 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7545-cbd733" event={"ID":"e84e1e08-7415-4d15-82a2-4ba4097ea8c0","Type":"ContainerStarted","Data":"7234edb454f8a3d051c054605e2e2441cee50a207f77dfcc6fa30cc635753bdb"} Sep 06 14:18:37.730485 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:37.730430 2550 generic.go:356] "Generic (PLEG): container finished" podID="e84e1e08-7415-4d15-82a2-4ba4097ea8c0" containerID="c2820d570638480d26baf7e38699faf2c9ddb2255bf727b8aa61b0b60520fa37" exitCode=0 Sep 06 14:18:37.730485 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:37.730487 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7545-cbd733" event={"ID":"e84e1e08-7415-4d15-82a2-4ba4097ea8c0","Type":"ContainerDied","Data":"c2820d570638480d26baf7e38699faf2c9ddb2255bf727b8aa61b0b60520fa37"} Sep 06 14:18:38.891528 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:38.891489 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:38.921134 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:38.921105 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7545-cbd733_e84e1e08-7415-4d15-82a2-4ba4097ea8c0/mt-curl-7545-cbd733/0.log" Sep 06 14:18:38.942815 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:38.942778 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7545-cbd733"] Sep 06 14:18:38.946187 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:38.946141 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7545-cbd733"] Sep 06 14:18:38.976244 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:38.976208 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7clmm\" (UniqueName: \"kubernetes.io/projected/e84e1e08-7415-4d15-82a2-4ba4097ea8c0-kube-api-access-7clmm\") pod \"e84e1e08-7415-4d15-82a2-4ba4097ea8c0\" (UID: \"e84e1e08-7415-4d15-82a2-4ba4097ea8c0\") " Sep 06 14:18:38.978275 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:38.978248 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84e1e08-7415-4d15-82a2-4ba4097ea8c0-kube-api-access-7clmm" (OuterVolumeSpecName: "kube-api-access-7clmm") pod "e84e1e08-7415-4d15-82a2-4ba4097ea8c0" (UID: "e84e1e08-7415-4d15-82a2-4ba4097ea8c0"). InnerVolumeSpecName "kube-api-access-7clmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:18:39.077774 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:39.077693 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7clmm\" (UniqueName: \"kubernetes.io/projected/e84e1e08-7415-4d15-82a2-4ba4097ea8c0-kube-api-access-7clmm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:18:39.236434 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:39.236403 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e84e1e08-7415-4d15-82a2-4ba4097ea8c0" path="/var/lib/kubelet/pods/e84e1e08-7415-4d15-82a2-4ba4097ea8c0/volumes" Sep 06 14:18:39.747968 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:39.747938 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-cbd733" Sep 06 14:18:39.747968 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:18:39.747961 2550 scope.go:117] "RemoveContainer" containerID="c2820d570638480d26baf7e38699faf2c9ddb2255bf727b8aa61b0b60520fa37" Sep 06 14:19:54.950689 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.950609 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8"] Sep 06 14:19:54.951104 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.951062 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84e1e08-7415-4d15-82a2-4ba4097ea8c0" containerName="mt-curl-7545-cbd733" Sep 06 14:19:54.951104 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.951074 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84e1e08-7415-4d15-82a2-4ba4097ea8c0" containerName="mt-curl-7545-cbd733" Sep 06 14:19:54.951176 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.951138 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e84e1e08-7415-4d15-82a2-4ba4097ea8c0" containerName="mt-curl-7545-cbd733" Sep 06 14:19:54.955305 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.955287 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:54.957570 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.957527 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-61d831-dockercfg-nmjk5\"" Sep 06 14:19:54.957713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.957569 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-61d831\"" Sep 06 14:19:54.962664 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:54.962637 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8"] Sep 06 14:19:55.003753 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.003721 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6139c482-b7a1-4065-837c-26ab603421b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.003753 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.003753 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nl7\" (UniqueName: \"kubernetes.io/projected/6139c482-b7a1-4065-837c-26ab603421b4-kube-api-access-n7nl7\") pod \"payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.104279 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.104243 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6139c482-b7a1-4065-837c-26ab603421b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.104279 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.104280 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nl7\" (UniqueName: \"kubernetes.io/projected/6139c482-b7a1-4065-837c-26ab603421b4-kube-api-access-n7nl7\") pod \"payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.105354 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.105173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6139c482-b7a1-4065-837c-26ab603421b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.111536 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.111508 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nl7\" (UniqueName: \"kubernetes.io/projected/6139c482-b7a1-4065-837c-26ab603421b4-kube-api-access-n7nl7\") pod \"payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.266985 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.266871 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:55.543556 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:55.543480 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8"] Sep 06 14:19:55.546800 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:19:55.546757 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6139c482_b7a1_4065_837c_26ab603421b4.slice/crio-30e5238f211350b87b11646e780f4510f042679f0f988ad94705b30ebbb14867 WatchSource:0}: Error finding container 30e5238f211350b87b11646e780f4510f042679f0f988ad94705b30ebbb14867: Status 404 returned error can't find the container with id 30e5238f211350b87b11646e780f4510f042679f0f988ad94705b30ebbb14867 Sep 06 14:19:56.230486 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:56.230441 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" event={"ID":"6139c482-b7a1-4065-837c-26ab603421b4","Type":"ContainerStarted","Data":"e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d"} Sep 06 14:19:56.230989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:56.230514 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" event={"ID":"6139c482-b7a1-4065-837c-26ab603421b4","Type":"ContainerStarted","Data":"30e5238f211350b87b11646e780f4510f042679f0f988ad94705b30ebbb14867"} Sep 06 14:19:56.230989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:56.230630 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:19:56.245791 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:19:56.245737 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" podStartSLOduration=2.245718815 podStartE2EDuration="2.245718815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:19:56.243918333 +0000 UTC m=+1345.507363058" watchObservedRunningTime="2026-09-06 14:19:56.245718815 +0000 UTC m=+1345.509163539" Sep 06 14:20:07.216381 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.216331 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7545-2dd40e"] Sep 06 14:20:07.220398 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.220366 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:07.222615 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.222590 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\"" Sep 06 14:20:07.223266 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.223238 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7545-2dd40e"] Sep 06 14:20:07.242535 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.242504 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:20:07.313911 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.313869 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fqp\" (UniqueName: \"kubernetes.io/projected/caf3a365-93bb-4f3f-ace3-d44b791a15f2-kube-api-access-s4fqp\") pod \"mt-curl-7545-2dd40e\" (UID: \"caf3a365-93bb-4f3f-ace3-d44b791a15f2\") " pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:07.415445 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.415404 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fqp\" (UniqueName: \"kubernetes.io/projected/caf3a365-93bb-4f3f-ace3-d44b791a15f2-kube-api-access-s4fqp\") pod \"mt-curl-7545-2dd40e\" (UID: \"caf3a365-93bb-4f3f-ace3-d44b791a15f2\") " pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:07.422734 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.422696 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fqp\" (UniqueName: \"kubernetes.io/projected/caf3a365-93bb-4f3f-ace3-d44b791a15f2-kube-api-access-s4fqp\") pod \"mt-curl-7545-2dd40e\" (UID: \"caf3a365-93bb-4f3f-ace3-d44b791a15f2\") " pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:07.533076 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.532972 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:07.808259 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:07.808234 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7545-2dd40e"] Sep 06 14:20:07.810267 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:20:07.810241 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaf3a365_93bb_4f3f_ace3_d44b791a15f2.slice/crio-32cb1af3c79216c30fdfe7f83668c744a74663749293409aa2e897257cf2cdf4 WatchSource:0}: Error finding container 32cb1af3c79216c30fdfe7f83668c744a74663749293409aa2e897257cf2cdf4: Status 404 returned error can't find the container with id 32cb1af3c79216c30fdfe7f83668c744a74663749293409aa2e897257cf2cdf4 Sep 06 14:20:08.310188 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:08.310152 2550 generic.go:356] "Generic (PLEG): container finished" podID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" containerID="ee08559e617029fc74f38c42a75deea21526dbc0c5737fb7463f65ca6855ba4d" exitCode=0 Sep 06 14:20:08.310572 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:08.310241 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7545-2dd40e" event={"ID":"caf3a365-93bb-4f3f-ace3-d44b791a15f2","Type":"ContainerDied","Data":"ee08559e617029fc74f38c42a75deea21526dbc0c5737fb7463f65ca6855ba4d"} Sep 06 14:20:08.310572 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:08.310287 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7545-2dd40e" event={"ID":"caf3a365-93bb-4f3f-ace3-d44b791a15f2","Type":"ContainerStarted","Data":"32cb1af3c79216c30fdfe7f83668c744a74663749293409aa2e897257cf2cdf4"} Sep 06 14:20:09.458670 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.458639 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:09.488444 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.488411 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7545-2dd40e_caf3a365-93bb-4f3f-ace3-d44b791a15f2/mt-curl-7545-2dd40e/0.log" Sep 06 14:20:09.508252 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.508209 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7545-2dd40e"] Sep 06 14:20:09.511350 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.511327 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7545-2dd40e"] Sep 06 14:20:09.541001 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.540970 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4fqp\" (UniqueName: \"kubernetes.io/projected/caf3a365-93bb-4f3f-ace3-d44b791a15f2-kube-api-access-s4fqp\") pod \"caf3a365-93bb-4f3f-ace3-d44b791a15f2\" (UID: \"caf3a365-93bb-4f3f-ace3-d44b791a15f2\") " Sep 06 14:20:09.543041 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.543019 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf3a365-93bb-4f3f-ace3-d44b791a15f2-kube-api-access-s4fqp" (OuterVolumeSpecName: "kube-api-access-s4fqp") pod "caf3a365-93bb-4f3f-ace3-d44b791a15f2" (UID: "caf3a365-93bb-4f3f-ace3-d44b791a15f2"). InnerVolumeSpecName "kube-api-access-s4fqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:20:09.620548 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.620517 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7545-580779"] Sep 06 14:20:09.620992 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.620977 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" containerName="mt-curl-7545-2dd40e" Sep 06 14:20:09.621042 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.620993 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" containerName="mt-curl-7545-2dd40e" Sep 06 14:20:09.621078 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.621062 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" containerName="mt-curl-7545-2dd40e" Sep 06 14:20:09.624304 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.624283 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:09.629304 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.629276 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7545-580779"] Sep 06 14:20:09.642137 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.642109 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4fqp\" (UniqueName: \"kubernetes.io/projected/caf3a365-93bb-4f3f-ace3-d44b791a15f2-kube-api-access-s4fqp\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:20:09.743069 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.742967 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94bd\" (UniqueName: \"kubernetes.io/projected/93175c08-c557-4251-a8cc-f8972039bdd8-kube-api-access-f94bd\") pod \"mt-curl-7545-580779\" (UID: \"93175c08-c557-4251-a8cc-f8972039bdd8\") " pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:09.843924 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.843869 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94bd\" (UniqueName: \"kubernetes.io/projected/93175c08-c557-4251-a8cc-f8972039bdd8-kube-api-access-f94bd\") pod \"mt-curl-7545-580779\" (UID: \"93175c08-c557-4251-a8cc-f8972039bdd8\") " pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:09.853129 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.853098 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94bd\" (UniqueName: \"kubernetes.io/projected/93175c08-c557-4251-a8cc-f8972039bdd8-kube-api-access-f94bd\") pod \"mt-curl-7545-580779\" (UID: \"93175c08-c557-4251-a8cc-f8972039bdd8\") " pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:09.936928 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:09.936852 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:10.211382 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:10.211275 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7545-580779"] Sep 06 14:20:10.214145 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:20:10.214120 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93175c08_c557_4251_a8cc_f8972039bdd8.slice/crio-f8244a35bf742068abce9724c4607fd4536c230bf5d91e02de75517be1c3b563 WatchSource:0}: Error finding container f8244a35bf742068abce9724c4607fd4536c230bf5d91e02de75517be1c3b563: Status 404 returned error can't find the container with id f8244a35bf742068abce9724c4607fd4536c230bf5d91e02de75517be1c3b563 Sep 06 14:20:10.324473 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:10.324429 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7545-580779" event={"ID":"93175c08-c557-4251-a8cc-f8972039bdd8","Type":"ContainerStarted","Data":"f8244a35bf742068abce9724c4607fd4536c230bf5d91e02de75517be1c3b563"} Sep 06 14:20:10.325726 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:10.325709 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:10.325838 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:10.325745 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32cb1af3c79216c30fdfe7f83668c744a74663749293409aa2e897257cf2cdf4" Sep 06 14:20:10.336112 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:20:10.336083 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7545-2dd40e\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:11.235513 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:20:11.235473 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7545-2dd40e\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" pod="opendatahub/mt-curl-7545-2dd40e" Sep 06 14:20:11.237587 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:11.237560 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf3a365-93bb-4f3f-ace3-d44b791a15f2" path="/var/lib/kubelet/pods/caf3a365-93bb-4f3f-ace3-d44b791a15f2/volumes" Sep 06 14:20:11.333528 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:11.333493 2550 generic.go:356] "Generic (PLEG): container finished" podID="93175c08-c557-4251-a8cc-f8972039bdd8" containerID="c0b0528975c3259e5d4f7882af8160f1c86f4b0d44a339f7e36442f47b218fc5" exitCode=0 Sep 06 14:20:11.333715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:11.333584 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7545-580779" event={"ID":"93175c08-c557-4251-a8cc-f8972039bdd8","Type":"ContainerDied","Data":"c0b0528975c3259e5d4f7882af8160f1c86f4b0d44a339f7e36442f47b218fc5"} Sep 06 14:20:12.477088 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.477056 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:12.494929 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.494883 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7545-580779_93175c08-c557-4251-a8cc-f8972039bdd8/mt-curl-7545-580779/0.log" Sep 06 14:20:12.519260 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.519225 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7545-580779"] Sep 06 14:20:12.523776 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.523729 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7545-580779"] Sep 06 14:20:12.571264 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.571223 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f94bd\" (UniqueName: \"kubernetes.io/projected/93175c08-c557-4251-a8cc-f8972039bdd8-kube-api-access-f94bd\") pod \"93175c08-c557-4251-a8cc-f8972039bdd8\" (UID: \"93175c08-c557-4251-a8cc-f8972039bdd8\") " Sep 06 14:20:12.573466 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.573432 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93175c08-c557-4251-a8cc-f8972039bdd8-kube-api-access-f94bd" (OuterVolumeSpecName: "kube-api-access-f94bd") pod "93175c08-c557-4251-a8cc-f8972039bdd8" (UID: "93175c08-c557-4251-a8cc-f8972039bdd8"). InnerVolumeSpecName "kube-api-access-f94bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:20:12.672538 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:12.672501 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f94bd\" (UniqueName: \"kubernetes.io/projected/93175c08-c557-4251-a8cc-f8972039bdd8-kube-api-access-f94bd\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:20:13.236656 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:13.236595 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93175c08-c557-4251-a8cc-f8972039bdd8" path="/var/lib/kubelet/pods/93175c08-c557-4251-a8cc-f8972039bdd8/volumes" Sep 06 14:20:13.348067 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:13.348041 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7545-580779" Sep 06 14:20:13.348259 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:13.348046 2550 scope.go:117] "RemoveContainer" containerID="c0b0528975c3259e5d4f7882af8160f1c86f4b0d44a339f7e36442f47b218fc5" Sep 06 14:20:22.687182 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.687110 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd"] Sep 06 14:20:22.687713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.687686 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93175c08-c557-4251-a8cc-f8972039bdd8" containerName="mt-curl-7545-580779" Sep 06 14:20:22.687713 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.687708 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="93175c08-c557-4251-a8cc-f8972039bdd8" containerName="mt-curl-7545-580779" Sep 06 14:20:22.687834 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.687822 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="93175c08-c557-4251-a8cc-f8972039bdd8" containerName="mt-curl-7545-580779" Sep 06 14:20:22.692483 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.692458 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.694993 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.694876 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9dd616-serving-cert\"" Sep 06 14:20:22.701744 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.701678 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd"] Sep 06 14:20:22.778358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.778310 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crcln\" (UniqueName: \"kubernetes.io/projected/bf618b57-dcac-4095-a15f-4b8d8933ca27-kube-api-access-crcln\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.778358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.778366 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf618b57-dcac-4095-a15f-4b8d8933ca27-openshift-service-ca\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.778604 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.778386 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.778604 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.778489 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-metrics-certs\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.879652 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.879603 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crcln\" (UniqueName: \"kubernetes.io/projected/bf618b57-dcac-4095-a15f-4b8d8933ca27-kube-api-access-crcln\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.879864 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.879665 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf618b57-dcac-4095-a15f-4b8d8933ca27-openshift-service-ca\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.879864 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.879704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.879864 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.879793 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-metrics-certs\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.879864 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:20:22.879829 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-serving-cert: secret "maas-api-e2e-api-b-9dd616-serving-cert" not found Sep 06 14:20:22.880066 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:20:22.879907 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls podName:bf618b57-dcac-4095-a15f-4b8d8933ca27 nodeName:}" failed. No retries permitted until 2026-09-06 14:20:23.379874239 +0000 UTC m=+1372.643318941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls") pod "maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" (UID: "bf618b57-dcac-4095-a15f-4b8d8933ca27") : secret "maas-api-e2e-api-b-9dd616-serving-cert" not found Sep 06 14:20:22.880347 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.880327 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf618b57-dcac-4095-a15f-4b8d8933ca27-openshift-service-ca\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.882296 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.882275 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-metrics-certs\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:22.887984 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:22.887959 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crcln\" (UniqueName: \"kubernetes.io/projected/bf618b57-dcac-4095-a15f-4b8d8933ca27-kube-api-access-crcln\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:23.384193 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:23.384157 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:23.386768 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:23.386744 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls\") pod \"maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:23.607199 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:23.607150 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:23.886013 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:23.885981 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd"] Sep 06 14:20:23.888012 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:20:23.887987 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf618b57_dcac_4095_a15f_4b8d8933ca27.slice/crio-0bf3e4edff3b868c2c33d74a66a5957e12294fd9053ba418ef4dda8d6db818dc WatchSource:0}: Error finding container 0bf3e4edff3b868c2c33d74a66a5957e12294fd9053ba418ef4dda8d6db818dc: Status 404 returned error can't find the container with id 0bf3e4edff3b868c2c33d74a66a5957e12294fd9053ba418ef4dda8d6db818dc Sep 06 14:20:24.429203 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:24.429164 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" event={"ID":"bf618b57-dcac-4095-a15f-4b8d8933ca27","Type":"ContainerStarted","Data":"5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9"} Sep 06 14:20:24.429388 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:24.429250 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" event={"ID":"bf618b57-dcac-4095-a15f-4b8d8933ca27","Type":"ContainerStarted","Data":"0bf3e4edff3b868c2c33d74a66a5957e12294fd9053ba418ef4dda8d6db818dc"} Sep 06 14:20:24.429436 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:24.429396 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:20:24.447342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:24.447269 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" podStartSLOduration=2.126543968 podStartE2EDuration="2.447242981s" totalImagesPullingTime="320.699013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:20:24.444698723 +0000 UTC m=+1373.708143451" watchObservedRunningTime="2026-09-06 14:20:24.447242981 +0000 UTC m=+1373.710687706" Sep 06 14:20:30.450385 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:20:30.450340 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:21:04.640056 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.640014 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8"] Sep 06 14:21:04.640632 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.640301 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" podUID="6139c482-b7a1-4065-837c-26ab603421b4" containerName="payload-processing" containerID="cri-o://e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d" gracePeriod=30 Sep 06 14:21:04.915572 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.915539 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:21:04.990937 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.990867 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7nl7\" (UniqueName: \"kubernetes.io/projected/6139c482-b7a1-4065-837c-26ab603421b4-kube-api-access-n7nl7\") pod \"6139c482-b7a1-4065-837c-26ab603421b4\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " Sep 06 14:21:04.991147 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.990975 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6139c482-b7a1-4065-837c-26ab603421b4-plugins-config-volume\") pod \"6139c482-b7a1-4065-837c-26ab603421b4\" (UID: \"6139c482-b7a1-4065-837c-26ab603421b4\") " Sep 06 14:21:04.991365 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.991339 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6139c482-b7a1-4065-837c-26ab603421b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6139c482-b7a1-4065-837c-26ab603421b4" (UID: "6139c482-b7a1-4065-837c-26ab603421b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:04.993177 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:04.993137 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6139c482-b7a1-4065-837c-26ab603421b4-kube-api-access-n7nl7" (OuterVolumeSpecName: "kube-api-access-n7nl7") pod "6139c482-b7a1-4065-837c-26ab603421b4" (UID: "6139c482-b7a1-4065-837c-26ab603421b4"). InnerVolumeSpecName "kube-api-access-n7nl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:05.092017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.091979 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7nl7\" (UniqueName: \"kubernetes.io/projected/6139c482-b7a1-4065-837c-26ab603421b4-kube-api-access-n7nl7\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:21:05.092017 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.092013 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6139c482-b7a1-4065-837c-26ab603421b4-plugins-config-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:21:05.704494 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.704453 2550 generic.go:356] "Generic (PLEG): container finished" podID="6139c482-b7a1-4065-837c-26ab603421b4" containerID="e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d" exitCode=0 Sep 06 14:21:05.704957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.704500 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" Sep 06 14:21:05.704957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.704543 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" event={"ID":"6139c482-b7a1-4065-837c-26ab603421b4","Type":"ContainerDied","Data":"e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d"} Sep 06 14:21:05.704957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.704604 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8" event={"ID":"6139c482-b7a1-4065-837c-26ab603421b4","Type":"ContainerDied","Data":"30e5238f211350b87b11646e780f4510f042679f0f988ad94705b30ebbb14867"} Sep 06 14:21:05.704957 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.704630 2550 scope.go:117] "RemoveContainer" containerID="e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d" Sep 06 14:21:05.714763 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.714741 2550 scope.go:117] "RemoveContainer" containerID="e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d" Sep 06 14:21:05.715304 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:21:05.715280 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d\": container with ID starting with e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d not found: ID does not exist" containerID="e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d" Sep 06 14:21:05.715426 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.715313 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d"} err="failed to get container status \"e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d\": rpc error: code = NotFound desc = could not find container \"e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d\": container with ID starting with e95648e72c91eef29099ee116905bf9ee5aad6bc849ed3d6232756ff8f0f3f8d not found: ID does not exist" Sep 06 14:21:05.720191 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.720159 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8"] Sep 06 14:21:05.725964 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:05.725927 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61d831-66f5c7d64f-tf8c8"] Sep 06 14:21:07.237393 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:07.237351 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6139c482-b7a1-4065-837c-26ab603421b4" path="/var/lib/kubelet/pods/6139c482-b7a1-4065-837c-26ab603421b4/volumes" Sep 06 14:21:27.288485 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.288444 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd"] Sep 06 14:21:27.289020 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.288692 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" podUID="bf618b57-dcac-4095-a15f-4b8d8933ca27" containerName="maas-api" containerID="cri-o://5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9" gracePeriod=30 Sep 06 14:21:27.554295 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.554222 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:21:27.603722 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.603678 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-metrics-certs\") pod \"bf618b57-dcac-4095-a15f-4b8d8933ca27\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " Sep 06 14:21:27.603926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.603783 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crcln\" (UniqueName: \"kubernetes.io/projected/bf618b57-dcac-4095-a15f-4b8d8933ca27-kube-api-access-crcln\") pod \"bf618b57-dcac-4095-a15f-4b8d8933ca27\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " Sep 06 14:21:27.603926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.603805 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf618b57-dcac-4095-a15f-4b8d8933ca27-openshift-service-ca\") pod \"bf618b57-dcac-4095-a15f-4b8d8933ca27\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " Sep 06 14:21:27.603926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.603828 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls\") pod \"bf618b57-dcac-4095-a15f-4b8d8933ca27\" (UID: \"bf618b57-dcac-4095-a15f-4b8d8933ca27\") " Sep 06 14:21:27.604249 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.604218 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf618b57-dcac-4095-a15f-4b8d8933ca27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf618b57-dcac-4095-a15f-4b8d8933ca27" (UID: "bf618b57-dcac-4095-a15f-4b8d8933ca27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:21:27.606116 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.606077 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf618b57-dcac-4095-a15f-4b8d8933ca27" (UID: "bf618b57-dcac-4095-a15f-4b8d8933ca27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:21:27.606255 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.606159 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf618b57-dcac-4095-a15f-4b8d8933ca27" (UID: "bf618b57-dcac-4095-a15f-4b8d8933ca27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:21:27.606255 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.606180 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf618b57-dcac-4095-a15f-4b8d8933ca27-kube-api-access-crcln" (OuterVolumeSpecName: "kube-api-access-crcln") pod "bf618b57-dcac-4095-a15f-4b8d8933ca27" (UID: "bf618b57-dcac-4095-a15f-4b8d8933ca27"). InnerVolumeSpecName "kube-api-access-crcln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:21:27.704547 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.704504 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:21:27.704547 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.704539 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crcln\" (UniqueName: \"kubernetes.io/projected/bf618b57-dcac-4095-a15f-4b8d8933ca27-kube-api-access-crcln\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:21:27.704547 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.704550 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf618b57-dcac-4095-a15f-4b8d8933ca27-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:21:27.704547 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.704558 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf618b57-dcac-4095-a15f-4b8d8933ca27-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:21:27.847190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.847092 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf618b57-dcac-4095-a15f-4b8d8933ca27" containerID="5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9" exitCode=0 Sep 06 14:21:27.847190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.847137 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" event={"ID":"bf618b57-dcac-4095-a15f-4b8d8933ca27","Type":"ContainerDied","Data":"5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9"} Sep 06 14:21:27.847190 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.847156 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" Sep 06 14:21:27.847598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.847214 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd" event={"ID":"bf618b57-dcac-4095-a15f-4b8d8933ca27","Type":"ContainerDied","Data":"0bf3e4edff3b868c2c33d74a66a5957e12294fd9053ba418ef4dda8d6db818dc"} Sep 06 14:21:27.847598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.847241 2550 scope.go:117] "RemoveContainer" containerID="5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9" Sep 06 14:21:27.857881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.857859 2550 scope.go:117] "RemoveContainer" containerID="5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9" Sep 06 14:21:27.858254 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:21:27.858232 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9\": container with ID starting with 5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9 not found: ID does not exist" containerID="5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9" Sep 06 14:21:27.858331 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.858262 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9"} err="failed to get container status \"5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9\": rpc error: code = NotFound desc = could not find container \"5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9\": container with ID starting with 5147de45a6199fa7750b231d9031207710793f245370dd350529573b8c1c12e9 not found: ID does not exist" Sep 06 14:21:27.868863 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.868824 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd"] Sep 06 14:21:27.873465 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:27.873429 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dd616-76d9bc5c49-9wpfd"] Sep 06 14:21:29.236679 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:21:29.236631 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf618b57-dcac-4095-a15f-4b8d8933ca27" path="/var/lib/kubelet/pods/bf618b57-dcac-4095-a15f-4b8d8933ca27/volumes" Sep 06 14:22:01.812679 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:01.812624 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6"] Sep 06 14:22:01.813352 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:01.812928 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" podUID="24344bb6-dccb-442a-9212-2e7a25b38654" containerName="maas-api" containerID="cri-o://869ea0b8ca3c7cd8b5ec15b9f4b7f5d171b6bc1e88c12bba7a6448bc3229d252" gracePeriod=30 Sep 06 14:22:01.877201 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:01.877164 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96"] Sep 06 14:22:01.877414 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:01.877391 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" podUID="b2d704db-29e0-45fa-91ad-98b2ca05ba85" containerName="payload-processing" containerID="cri-o://63274143fe007b232e87ed7f4985f1dfadefc7b4da4033bba9829c27bfb8e113" gracePeriod=30 Sep 06 14:22:02.073918 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.073796 2550 generic.go:356] "Generic (PLEG): container finished" podID="b2d704db-29e0-45fa-91ad-98b2ca05ba85" containerID="63274143fe007b232e87ed7f4985f1dfadefc7b4da4033bba9829c27bfb8e113" exitCode=0 Sep 06 14:22:02.073918 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.073849 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" event={"ID":"b2d704db-29e0-45fa-91ad-98b2ca05ba85","Type":"ContainerDied","Data":"63274143fe007b232e87ed7f4985f1dfadefc7b4da4033bba9829c27bfb8e113"} Sep 06 14:22:02.076856 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.076820 2550 generic.go:356] "Generic (PLEG): container finished" podID="24344bb6-dccb-442a-9212-2e7a25b38654" containerID="869ea0b8ca3c7cd8b5ec15b9f4b7f5d171b6bc1e88c12bba7a6448bc3229d252" exitCode=0 Sep 06 14:22:02.077051 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.077001 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" event={"ID":"24344bb6-dccb-442a-9212-2e7a25b38654","Type":"ContainerDied","Data":"869ea0b8ca3c7cd8b5ec15b9f4b7f5d171b6bc1e88c12bba7a6448bc3229d252"} Sep 06 14:22:02.148670 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.148643 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:22:02.184306 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.184267 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:22:02.336018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.335883 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z792m\" (UniqueName: \"kubernetes.io/projected/24344bb6-dccb-442a-9212-2e7a25b38654-kube-api-access-z792m\") pod \"24344bb6-dccb-442a-9212-2e7a25b38654\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " Sep 06 14:22:02.336018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.335980 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24344bb6-dccb-442a-9212-2e7a25b38654-openshift-service-ca\") pod \"24344bb6-dccb-442a-9212-2e7a25b38654\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " Sep 06 14:22:02.336018 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.336009 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nh8h\" (UniqueName: \"kubernetes.io/projected/b2d704db-29e0-45fa-91ad-98b2ca05ba85-kube-api-access-5nh8h\") pod \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " Sep 06 14:22:02.336283 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.336065 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-metrics-certs\") pod \"24344bb6-dccb-442a-9212-2e7a25b38654\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " Sep 06 14:22:02.336283 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.336082 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d704db-29e0-45fa-91ad-98b2ca05ba85-plugins-config-volume\") pod \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\" (UID: \"b2d704db-29e0-45fa-91ad-98b2ca05ba85\") " Sep 06 14:22:02.336283 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.336115 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls\") pod \"24344bb6-dccb-442a-9212-2e7a25b38654\" (UID: \"24344bb6-dccb-442a-9212-2e7a25b38654\") " Sep 06 14:22:02.336454 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.336415 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24344bb6-dccb-442a-9212-2e7a25b38654-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24344bb6-dccb-442a-9212-2e7a25b38654" (UID: "24344bb6-dccb-442a-9212-2e7a25b38654"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:22:02.336515 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.336481 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d704db-29e0-45fa-91ad-98b2ca05ba85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2d704db-29e0-45fa-91ad-98b2ca05ba85" (UID: "b2d704db-29e0-45fa-91ad-98b2ca05ba85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:22:02.338698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.338657 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24344bb6-dccb-442a-9212-2e7a25b38654" (UID: "24344bb6-dccb-442a-9212-2e7a25b38654"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:22:02.338698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.338648 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24344bb6-dccb-442a-9212-2e7a25b38654" (UID: "24344bb6-dccb-442a-9212-2e7a25b38654"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:22:02.338907 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.338794 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24344bb6-dccb-442a-9212-2e7a25b38654-kube-api-access-z792m" (OuterVolumeSpecName: "kube-api-access-z792m") pod "24344bb6-dccb-442a-9212-2e7a25b38654" (UID: "24344bb6-dccb-442a-9212-2e7a25b38654"). InnerVolumeSpecName "kube-api-access-z792m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:22:02.338907 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.338856 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d704db-29e0-45fa-91ad-98b2ca05ba85-kube-api-access-5nh8h" (OuterVolumeSpecName: "kube-api-access-5nh8h") pod "b2d704db-29e0-45fa-91ad-98b2ca05ba85" (UID: "b2d704db-29e0-45fa-91ad-98b2ca05ba85"). InnerVolumeSpecName "kube-api-access-5nh8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:22:02.437435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.437387 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.437435 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.437435 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z792m\" (UniqueName: \"kubernetes.io/projected/24344bb6-dccb-442a-9212-2e7a25b38654-kube-api-access-z792m\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.437723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.437451 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24344bb6-dccb-442a-9212-2e7a25b38654-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.437723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.437464 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nh8h\" (UniqueName: \"kubernetes.io/projected/b2d704db-29e0-45fa-91ad-98b2ca05ba85-kube-api-access-5nh8h\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.437723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.437477 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24344bb6-dccb-442a-9212-2e7a25b38654-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:22:02.437723 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:02.437490 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d704db-29e0-45fa-91ad-98b2ca05ba85-plugins-config-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:22:03.084643 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.084543 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" event={"ID":"b2d704db-29e0-45fa-91ad-98b2ca05ba85","Type":"ContainerDied","Data":"a4f3b73b2897f5d42a8a12a2d4c54e3f604c7a3fded68878cbbc96cf474e8d8f"} Sep 06 14:22:03.084643 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.084562 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96" Sep 06 14:22:03.085551 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.084629 2550 scope.go:117] "RemoveContainer" containerID="63274143fe007b232e87ed7f4985f1dfadefc7b4da4033bba9829c27bfb8e113" Sep 06 14:22:03.086266 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.086191 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" event={"ID":"24344bb6-dccb-442a-9212-2e7a25b38654","Type":"ContainerDied","Data":"66c351333004ea384b4ceb53f80be9f658859d4af967c011e3ee375aa465855e"} Sep 06 14:22:03.086266 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.086198 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6" Sep 06 14:22:03.095722 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.095697 2550 scope.go:117] "RemoveContainer" containerID="869ea0b8ca3c7cd8b5ec15b9f4b7f5d171b6bc1e88c12bba7a6448bc3229d252" Sep 06 14:22:03.110100 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.110064 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6"] Sep 06 14:22:03.115799 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.115763 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a388e1-6cc4c8495b-bxjq6"] Sep 06 14:22:03.125985 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.125946 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96"] Sep 06 14:22:03.128535 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.128508 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a388e1-756bc56f5f-7kn96"] Sep 06 14:22:03.237212 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.237164 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24344bb6-dccb-442a-9212-2e7a25b38654" path="/var/lib/kubelet/pods/24344bb6-dccb-442a-9212-2e7a25b38654/volumes" Sep 06 14:22:03.237655 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:03.237633 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d704db-29e0-45fa-91ad-98b2ca05ba85" path="/var/lib/kubelet/pods/b2d704db-29e0-45fa-91ad-98b2ca05ba85/volumes" Sep 06 14:22:31.339201 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:31.339172 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:22:31.345441 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:31.345413 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:22:51.509765 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.509725 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t"] Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510190 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24344bb6-dccb-442a-9212-2e7a25b38654" containerName="maas-api" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510203 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="24344bb6-dccb-442a-9212-2e7a25b38654" containerName="maas-api" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510215 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6139c482-b7a1-4065-837c-26ab603421b4" containerName="payload-processing" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510221 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6139c482-b7a1-4065-837c-26ab603421b4" containerName="payload-processing" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510241 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d704db-29e0-45fa-91ad-98b2ca05ba85" containerName="payload-processing" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510247 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d704db-29e0-45fa-91ad-98b2ca05ba85" containerName="payload-processing" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510258 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf618b57-dcac-4095-a15f-4b8d8933ca27" containerName="maas-api" Sep 06 14:22:51.510270 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510263 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf618b57-dcac-4095-a15f-4b8d8933ca27" containerName="maas-api" Sep 06 14:22:51.510529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510347 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d704db-29e0-45fa-91ad-98b2ca05ba85" containerName="payload-processing" Sep 06 14:22:51.510529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510357 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf618b57-dcac-4095-a15f-4b8d8933ca27" containerName="maas-api" Sep 06 14:22:51.510529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510364 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6139c482-b7a1-4065-837c-26ab603421b4" containerName="payload-processing" Sep 06 14:22:51.510529 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.510370 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="24344bb6-dccb-442a-9212-2e7a25b38654" containerName="maas-api" Sep 06 14:22:51.515180 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.515153 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.517419 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.517382 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c1a83e-serving-cert\"" Sep 06 14:22:51.525630 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.525346 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t"] Sep 06 14:22:51.589092 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.589059 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-metrics-certs\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.589311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.589157 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.589311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.589264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493ae2c-7d48-4ed2-89ee-6e355727daf7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.589311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.589295 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gnh\" (UniqueName: \"kubernetes.io/projected/8493ae2c-7d48-4ed2-89ee-6e355727daf7-kube-api-access-v5gnh\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.690354 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.690309 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-metrics-certs\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.690553 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.690373 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.690553 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.690426 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493ae2c-7d48-4ed2-89ee-6e355727daf7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.690553 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.690447 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gnh\" (UniqueName: \"kubernetes.io/projected/8493ae2c-7d48-4ed2-89ee-6e355727daf7-kube-api-access-v5gnh\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.690726 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:22:51.690635 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-serving-cert: secret "maas-api-e2e-ipp-b-c1a83e-serving-cert" not found Sep 06 14:22:51.690786 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:22:51.690736 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls podName:8493ae2c-7d48-4ed2-89ee-6e355727daf7 nodeName:}" failed. No retries permitted until 2026-09-06 14:22:52.190698027 +0000 UTC m=+1521.454142738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls") pod "maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" (UID: "8493ae2c-7d48-4ed2-89ee-6e355727daf7") : secret "maas-api-e2e-ipp-b-c1a83e-serving-cert" not found Sep 06 14:22:51.691123 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.691106 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493ae2c-7d48-4ed2-89ee-6e355727daf7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.692960 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.692929 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-metrics-certs\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:51.698580 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:51.698554 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gnh\" (UniqueName: \"kubernetes.io/projected/8493ae2c-7d48-4ed2-89ee-6e355727daf7-kube-api-access-v5gnh\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:52.194399 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:52.194362 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:52.196953 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:52.196917 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:52.430430 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:52.430395 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:52.721484 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:52.721457 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t"] Sep 06 14:22:52.723766 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:22:52.723734 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8493ae2c_7d48_4ed2_89ee_6e355727daf7.slice/crio-e5ad9bc4431dfa3377af976e35723852ac0d07e8319e414e991e1f994eaecc7e WatchSource:0}: Error finding container e5ad9bc4431dfa3377af976e35723852ac0d07e8319e414e991e1f994eaecc7e: Status 404 returned error can't find the container with id e5ad9bc4431dfa3377af976e35723852ac0d07e8319e414e991e1f994eaecc7e Sep 06 14:22:53.407046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:53.406921 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" event={"ID":"8493ae2c-7d48-4ed2-89ee-6e355727daf7","Type":"ContainerStarted","Data":"88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a"} Sep 06 14:22:53.407046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:53.406981 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" event={"ID":"8493ae2c-7d48-4ed2-89ee-6e355727daf7","Type":"ContainerStarted","Data":"e5ad9bc4431dfa3377af976e35723852ac0d07e8319e414e991e1f994eaecc7e"} Sep 06 14:22:53.407046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:53.407035 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:22:53.426979 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:53.426924 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" podStartSLOduration=2.096805356 podStartE2EDuration="2.42688776s" totalImagesPullingTime="330.082404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:53.423376607 +0000 UTC m=+1522.686821331" watchObservedRunningTime="2026-09-06 14:22:53.42688776 +0000 UTC m=+1522.690332484" Sep 06 14:22:59.419637 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:22:59.419605 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:23:31.539805 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:23:31.539767 2550 scope.go:117] "RemoveContainer" containerID="d064fc1a0bb871b43b8ee1c2ebe1b68c0357e55100dbe23ac5bf16cfada0a935" Sep 06 14:24:05.803984 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.803937 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6"] Sep 06 14:24:05.808392 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.808366 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.811226 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.811205 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8e268e-serving-cert\"" Sep 06 14:24:05.817958 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.817862 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6"] Sep 06 14:24:05.853711 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.853667 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.853939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.853735 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.853939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.853773 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928hr\" (UniqueName: \"kubernetes.io/projected/0d341388-a8a3-4b1e-bf26-49be8f196d89-kube-api-access-928hr\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.853939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.853795 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d341388-a8a3-4b1e-bf26-49be8f196d89-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.955197 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.955161 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.955197 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.955214 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.955488 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.955238 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928hr\" (UniqueName: \"kubernetes.io/projected/0d341388-a8a3-4b1e-bf26-49be8f196d89-kube-api-access-928hr\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.955488 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.955258 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d341388-a8a3-4b1e-bf26-49be8f196d89-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.956127 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.956030 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d341388-a8a3-4b1e-bf26-49be8f196d89-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.957845 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.957820 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.957952 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.957843 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:05.965410 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:05.965379 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-928hr\" (UniqueName: \"kubernetes.io/projected/0d341388-a8a3-4b1e-bf26-49be8f196d89-kube-api-access-928hr\") pod \"maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:06.123371 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.123270 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:06.410473 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.410443 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6"] Sep 06 14:24:06.411940 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:24:06.411877 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d341388_a8a3_4b1e_bf26_49be8f196d89.slice/crio-3751ef4a9cc35f387daec2e093142ef9eea8846ff52949c349346993415055eb WatchSource:0}: Error finding container 3751ef4a9cc35f387daec2e093142ef9eea8846ff52949c349346993415055eb: Status 404 returned error can't find the container with id 3751ef4a9cc35f387daec2e093142ef9eea8846ff52949c349346993415055eb Sep 06 14:24:06.413417 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.413398 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:24:06.878965 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.878929 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" event={"ID":"0d341388-a8a3-4b1e-bf26-49be8f196d89","Type":"ContainerStarted","Data":"b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29"} Sep 06 14:24:06.879416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.878988 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" event={"ID":"0d341388-a8a3-4b1e-bf26-49be8f196d89","Type":"ContainerStarted","Data":"3751ef4a9cc35f387daec2e093142ef9eea8846ff52949c349346993415055eb"} Sep 06 14:24:06.879416 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.879021 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:06.898274 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:06.898214 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" podStartSLOduration=1.5305342020000001 podStartE2EDuration="1.898192389s" totalImagesPullingTime="367.658187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:24:06.896012333 +0000 UTC m=+1596.159457057" watchObservedRunningTime="2026-09-06 14:24:06.898192389 +0000 UTC m=+1596.161637158" Sep 06 14:24:12.891406 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:12.891370 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:36.050237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.049915 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6"] Sep 06 14:24:36.050237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.050158 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" podUID="0d341388-a8a3-4b1e-bf26-49be8f196d89" containerName="maas-api" containerID="cri-o://b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29" gracePeriod=30 Sep 06 14:24:36.322598 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.322564 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:36.447970 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.447934 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-928hr\" (UniqueName: \"kubernetes.io/projected/0d341388-a8a3-4b1e-bf26-49be8f196d89-kube-api-access-928hr\") pod \"0d341388-a8a3-4b1e-bf26-49be8f196d89\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " Sep 06 14:24:36.448155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.447982 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-maas-api-tls\") pod \"0d341388-a8a3-4b1e-bf26-49be8f196d89\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " Sep 06 14:24:36.448155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.448011 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d341388-a8a3-4b1e-bf26-49be8f196d89-openshift-service-ca\") pod \"0d341388-a8a3-4b1e-bf26-49be8f196d89\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " Sep 06 14:24:36.448155 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.448073 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-metrics-certs\") pod \"0d341388-a8a3-4b1e-bf26-49be8f196d89\" (UID: \"0d341388-a8a3-4b1e-bf26-49be8f196d89\") " Sep 06 14:24:36.448394 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.448364 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d341388-a8a3-4b1e-bf26-49be8f196d89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d341388-a8a3-4b1e-bf26-49be8f196d89" (UID: "0d341388-a8a3-4b1e-bf26-49be8f196d89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:24:36.450318 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.450290 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d341388-a8a3-4b1e-bf26-49be8f196d89-kube-api-access-928hr" (OuterVolumeSpecName: "kube-api-access-928hr") pod "0d341388-a8a3-4b1e-bf26-49be8f196d89" (UID: "0d341388-a8a3-4b1e-bf26-49be8f196d89"). InnerVolumeSpecName "kube-api-access-928hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:24:36.450318 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.450304 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d341388-a8a3-4b1e-bf26-49be8f196d89" (UID: "0d341388-a8a3-4b1e-bf26-49be8f196d89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:24:36.450446 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.450385 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d341388-a8a3-4b1e-bf26-49be8f196d89" (UID: "0d341388-a8a3-4b1e-bf26-49be8f196d89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:24:36.549094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.549055 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:24:36.549094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.549087 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-928hr\" (UniqueName: \"kubernetes.io/projected/0d341388-a8a3-4b1e-bf26-49be8f196d89-kube-api-access-928hr\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:24:36.549094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.549098 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d341388-a8a3-4b1e-bf26-49be8f196d89-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:24:36.549366 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:36.549107 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d341388-a8a3-4b1e-bf26-49be8f196d89-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:24:37.074310 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.074275 2550 generic.go:356] "Generic (PLEG): container finished" podID="0d341388-a8a3-4b1e-bf26-49be8f196d89" containerID="b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29" exitCode=0 Sep 06 14:24:37.074930 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.074327 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" Sep 06 14:24:37.074930 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.074339 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" event={"ID":"0d341388-a8a3-4b1e-bf26-49be8f196d89","Type":"ContainerDied","Data":"b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29"} Sep 06 14:24:37.074930 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.074382 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6" event={"ID":"0d341388-a8a3-4b1e-bf26-49be8f196d89","Type":"ContainerDied","Data":"3751ef4a9cc35f387daec2e093142ef9eea8846ff52949c349346993415055eb"} Sep 06 14:24:37.074930 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.074404 2550 scope.go:117] "RemoveContainer" containerID="b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29" Sep 06 14:24:37.084859 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.084828 2550 scope.go:117] "RemoveContainer" containerID="b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29" Sep 06 14:24:37.085258 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:24:37.085234 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29\": container with ID starting with b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29 not found: ID does not exist" containerID="b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29" Sep 06 14:24:37.085368 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.085265 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29"} err="failed to get container status \"b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29\": rpc error: code = NotFound desc = could not find container \"b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29\": container with ID starting with b4b9339ea03e7bf5996690a6e0321fbd0698dfd9af998eac104d4b57100bfa29 not found: ID does not exist" Sep 06 14:24:37.096697 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.096656 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6"] Sep 06 14:24:37.099282 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.099243 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e268e-7898498b56-8cpz6"] Sep 06 14:24:37.236693 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:24:37.236659 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d341388-a8a3-4b1e-bf26-49be8f196d89" path="/var/lib/kubelet/pods/0d341388-a8a3-4b1e-bf26-49be8f196d89/volumes" Sep 06 14:26:00.941297 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:00.941202 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t"] Sep 06 14:26:00.942258 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:00.942218 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" podUID="8493ae2c-7d48-4ed2-89ee-6e355727daf7" containerName="maas-api" containerID="cri-o://88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a" gracePeriod=30 Sep 06 14:26:01.216045 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.216011 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:26:01.347617 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.347576 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls\") pod \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " Sep 06 14:26:01.347617 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.347625 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-metrics-certs\") pod \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " Sep 06 14:26:01.347861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.347670 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5gnh\" (UniqueName: \"kubernetes.io/projected/8493ae2c-7d48-4ed2-89ee-6e355727daf7-kube-api-access-v5gnh\") pod \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " Sep 06 14:26:01.347861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.347742 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493ae2c-7d48-4ed2-89ee-6e355727daf7-openshift-service-ca\") pod \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\" (UID: \"8493ae2c-7d48-4ed2-89ee-6e355727daf7\") " Sep 06 14:26:01.348243 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.348195 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8493ae2c-7d48-4ed2-89ee-6e355727daf7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8493ae2c-7d48-4ed2-89ee-6e355727daf7" (UID: "8493ae2c-7d48-4ed2-89ee-6e355727daf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:01.349959 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.349931 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8493ae2c-7d48-4ed2-89ee-6e355727daf7" (UID: "8493ae2c-7d48-4ed2-89ee-6e355727daf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:01.350094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.349973 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8493ae2c-7d48-4ed2-89ee-6e355727daf7-kube-api-access-v5gnh" (OuterVolumeSpecName: "kube-api-access-v5gnh") pod "8493ae2c-7d48-4ed2-89ee-6e355727daf7" (UID: "8493ae2c-7d48-4ed2-89ee-6e355727daf7"). InnerVolumeSpecName "kube-api-access-v5gnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:01.350094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.349995 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8493ae2c-7d48-4ed2-89ee-6e355727daf7" (UID: "8493ae2c-7d48-4ed2-89ee-6e355727daf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:01.448861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.448761 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:01.448861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.448795 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493ae2c-7d48-4ed2-89ee-6e355727daf7-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:01.448861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.448809 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gnh\" (UniqueName: \"kubernetes.io/projected/8493ae2c-7d48-4ed2-89ee-6e355727daf7-kube-api-access-v5gnh\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:01.448861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.448822 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493ae2c-7d48-4ed2-89ee-6e355727daf7-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:01.597510 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.597475 2550 generic.go:356] "Generic (PLEG): container finished" podID="8493ae2c-7d48-4ed2-89ee-6e355727daf7" containerID="88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a" exitCode=0 Sep 06 14:26:01.597510 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.597523 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" Sep 06 14:26:01.597732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.597561 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" event={"ID":"8493ae2c-7d48-4ed2-89ee-6e355727daf7","Type":"ContainerDied","Data":"88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a"} Sep 06 14:26:01.597732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.597622 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t" event={"ID":"8493ae2c-7d48-4ed2-89ee-6e355727daf7","Type":"ContainerDied","Data":"e5ad9bc4431dfa3377af976e35723852ac0d07e8319e414e991e1f994eaecc7e"} Sep 06 14:26:01.597732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.597644 2550 scope.go:117] "RemoveContainer" containerID="88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a" Sep 06 14:26:01.608004 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.607982 2550 scope.go:117] "RemoveContainer" containerID="88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a" Sep 06 14:26:01.608364 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:26:01.608341 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a\": container with ID starting with 88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a not found: ID does not exist" containerID="88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a" Sep 06 14:26:01.608412 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.608373 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a"} err="failed to get container status \"88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a\": rpc error: code = NotFound desc = could not find container \"88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a\": container with ID starting with 88b6955a05614e3af1958f262f40e6847fb7169b259446aaeedacbbe9dfe3b3a not found: ID does not exist" Sep 06 14:26:01.619181 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.619142 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t"] Sep 06 14:26:01.623205 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:01.623167 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c1a83e-866f6f587c-wbw2t"] Sep 06 14:26:03.236539 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:03.236496 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8493ae2c-7d48-4ed2-89ee-6e355727daf7" path="/var/lib/kubelet/pods/8493ae2c-7d48-4ed2-89ee-6e355727daf7/volumes" Sep 06 14:26:18.250412 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.250369 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn"] Sep 06 14:26:18.250915 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.250634 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" podUID="63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" containerName="maas-api" containerID="cri-o://4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58" gracePeriod=30 Sep 06 14:26:18.294262 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.294226 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q"] Sep 06 14:26:18.294528 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.294485 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" podUID="ff39c708-38d5-4084-aa9f-dfa8d3241cdd" containerName="payload-processing" containerID="cri-o://6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab" gracePeriod=30 Sep 06 14:26:18.554923 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.554882 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:26:18.596939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.596884 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:26:18.603122 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603075 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls\") pod \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " Sep 06 14:26:18.603311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603138 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-metrics-certs\") pod \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " Sep 06 14:26:18.603311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603157 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g7l4\" (UniqueName: \"kubernetes.io/projected/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-kube-api-access-9g7l4\") pod \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " Sep 06 14:26:18.603311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603186 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-plugins-config-volume\") pod \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\" (UID: \"ff39c708-38d5-4084-aa9f-dfa8d3241cdd\") " Sep 06 14:26:18.603311 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603205 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-openshift-service-ca\") pod \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " Sep 06 14:26:18.603795 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603734 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff39c708-38d5-4084-aa9f-dfa8d3241cdd" (UID: "ff39c708-38d5-4084-aa9f-dfa8d3241cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:18.603795 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.603745 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" (UID: "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:18.605654 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.605617 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" (UID: "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:18.605869 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.605847 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" (UID: "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:18.605950 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.605869 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-kube-api-access-9g7l4" (OuterVolumeSpecName: "kube-api-access-9g7l4") pod "ff39c708-38d5-4084-aa9f-dfa8d3241cdd" (UID: "ff39c708-38d5-4084-aa9f-dfa8d3241cdd"). InnerVolumeSpecName "kube-api-access-9g7l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:18.703859 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.703823 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pwn8\" (UniqueName: \"kubernetes.io/projected/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-kube-api-access-2pwn8\") pod \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\" (UID: \"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c\") " Sep 06 14:26:18.704073 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.704056 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.704168 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.704076 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.704168 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.704091 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7l4\" (UniqueName: \"kubernetes.io/projected/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-kube-api-access-9g7l4\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.704168 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.704106 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff39c708-38d5-4084-aa9f-dfa8d3241cdd-plugins-config-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.704168 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.704117 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:18.705353 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.705327 2550 generic.go:356] "Generic (PLEG): container finished" podID="63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" containerID="4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58" exitCode=0 Sep 06 14:26:18.705489 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.705396 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" Sep 06 14:26:18.705489 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.705424 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" event={"ID":"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c","Type":"ContainerDied","Data":"4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58"} Sep 06 14:26:18.705606 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.705496 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn" event={"ID":"63fe9f6d-2c01-44bb-9ba7-32f2ca10603c","Type":"ContainerDied","Data":"29def9ba3eaeea2f9f691157640378cb5d7a85ee77afdb335eeecad31069f0bb"} Sep 06 14:26:18.705606 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.705528 2550 scope.go:117] "RemoveContainer" containerID="4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58" Sep 06 14:26:18.706310 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.706287 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-kube-api-access-2pwn8" (OuterVolumeSpecName: "kube-api-access-2pwn8") pod "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" (UID: "63fe9f6d-2c01-44bb-9ba7-32f2ca10603c"). InnerVolumeSpecName "kube-api-access-2pwn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:18.706886 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.706868 2550 generic.go:356] "Generic (PLEG): container finished" podID="ff39c708-38d5-4084-aa9f-dfa8d3241cdd" containerID="6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab" exitCode=0 Sep 06 14:26:18.706997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.706910 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" event={"ID":"ff39c708-38d5-4084-aa9f-dfa8d3241cdd","Type":"ContainerDied","Data":"6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab"} Sep 06 14:26:18.706997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.706934 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" Sep 06 14:26:18.706997 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.706960 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q" event={"ID":"ff39c708-38d5-4084-aa9f-dfa8d3241cdd","Type":"ContainerDied","Data":"34994114026f8190a777a468b424fca79fbd04331620cb6c0e8eb38236889353"} Sep 06 14:26:18.715930 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.715878 2550 scope.go:117] "RemoveContainer" containerID="4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58" Sep 06 14:26:18.716269 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:26:18.716249 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58\": container with ID starting with 4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58 not found: ID does not exist" containerID="4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58" Sep 06 14:26:18.716346 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.716283 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58"} err="failed to get container status \"4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58\": rpc error: code = NotFound desc = could not find container \"4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58\": container with ID starting with 4e6264db5351566a6d3b8139c5830fa086c5fb35d36728ad33c8d8391d947c58 not found: ID does not exist" Sep 06 14:26:18.716346 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.716307 2550 scope.go:117] "RemoveContainer" containerID="6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab" Sep 06 14:26:18.725351 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.725328 2550 scope.go:117] "RemoveContainer" containerID="6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab" Sep 06 14:26:18.725699 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:26:18.725676 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab\": container with ID starting with 6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab not found: ID does not exist" containerID="6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab" Sep 06 14:26:18.725767 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.725709 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab"} err="failed to get container status \"6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab\": rpc error: code = NotFound desc = could not find container \"6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab\": container with ID starting with 6ee7e03b3bcbccfd1b69265a773d865da96156e3705afbc035b430ce4d8f04ab not found: ID does not exist" Sep 06 14:26:18.729586 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.729553 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q"] Sep 06 14:26:18.735013 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.734978 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0aaff3-8597854d8f-gh87q"] Sep 06 14:26:18.804720 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:18.804635 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pwn8\" (UniqueName: \"kubernetes.io/projected/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c-kube-api-access-2pwn8\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:19.028391 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:19.028347 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn"] Sep 06 14:26:19.033191 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:19.033153 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0aaff3-867d487759-vzrkn"] Sep 06 14:26:19.236477 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:19.236433 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" path="/var/lib/kubelet/pods/63fe9f6d-2c01-44bb-9ba7-32f2ca10603c/volumes" Sep 06 14:26:19.236933 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:19.236913 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff39c708-38d5-4084-aa9f-dfa8d3241cdd" path="/var/lib/kubelet/pods/ff39c708-38d5-4084-aa9f-dfa8d3241cdd/volumes" Sep 06 14:26:31.589211 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:31.589174 2550 scope.go:117] "RemoveContainer" containerID="ee08559e617029fc74f38c42a75deea21526dbc0c5737fb7463f65ca6855ba4d" Sep 06 14:26:35.557380 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.557342 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz"] Sep 06 14:26:35.558240 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.558207 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" podUID="a11127b7-5959-444e-af44-247ad72bd39a" containerName="maas-api" containerID="cri-o://0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906" gracePeriod=30 Sep 06 14:26:35.810850 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.810768 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:26:35.811965 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.811939 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" event={"ID":"a11127b7-5959-444e-af44-247ad72bd39a","Type":"ContainerDied","Data":"0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906"} Sep 06 14:26:35.811965 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.811880 2550 generic.go:356] "Generic (PLEG): container finished" podID="a11127b7-5959-444e-af44-247ad72bd39a" containerID="0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906" exitCode=0 Sep 06 14:26:35.812116 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.812005 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" event={"ID":"a11127b7-5959-444e-af44-247ad72bd39a","Type":"ContainerDied","Data":"1c73d322de49a7024e9a5e3cf79d06ae24856b5f79bcfc8150748fe3ccba4897"} Sep 06 14:26:35.812116 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.812034 2550 scope.go:117] "RemoveContainer" containerID="0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906" Sep 06 14:26:35.823316 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.823294 2550 scope.go:117] "RemoveContainer" containerID="0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906" Sep 06 14:26:35.823755 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:26:35.823729 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906\": container with ID starting with 0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906 not found: ID does not exist" containerID="0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906" Sep 06 14:26:35.823852 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.823768 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906"} err="failed to get container status \"0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906\": rpc error: code = NotFound desc = could not find container \"0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906\": container with ID starting with 0aa1b5a6e32daa4f9b6f43d1f36784f19ad570a033c8d1db6f174c993581a906 not found: ID does not exist" Sep 06 14:26:35.856859 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.856820 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlgbn\" (UniqueName: \"kubernetes.io/projected/a11127b7-5959-444e-af44-247ad72bd39a-kube-api-access-vlgbn\") pod \"a11127b7-5959-444e-af44-247ad72bd39a\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " Sep 06 14:26:35.857060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.856915 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls\") pod \"a11127b7-5959-444e-af44-247ad72bd39a\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " Sep 06 14:26:35.857060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.856949 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-metrics-certs\") pod \"a11127b7-5959-444e-af44-247ad72bd39a\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " Sep 06 14:26:35.857060 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.857000 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11127b7-5959-444e-af44-247ad72bd39a-openshift-service-ca\") pod \"a11127b7-5959-444e-af44-247ad72bd39a\" (UID: \"a11127b7-5959-444e-af44-247ad72bd39a\") " Sep 06 14:26:35.857484 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.857449 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11127b7-5959-444e-af44-247ad72bd39a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a11127b7-5959-444e-af44-247ad72bd39a" (UID: "a11127b7-5959-444e-af44-247ad72bd39a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:26:35.857773 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.857751 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11127b7-5959-444e-af44-247ad72bd39a-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:35.859118 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.859092 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a11127b7-5959-444e-af44-247ad72bd39a" (UID: "a11127b7-5959-444e-af44-247ad72bd39a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:35.859590 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.859566 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a11127b7-5959-444e-af44-247ad72bd39a" (UID: "a11127b7-5959-444e-af44-247ad72bd39a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:35.859681 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.859568 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11127b7-5959-444e-af44-247ad72bd39a-kube-api-access-vlgbn" (OuterVolumeSpecName: "kube-api-access-vlgbn") pod "a11127b7-5959-444e-af44-247ad72bd39a" (UID: "a11127b7-5959-444e-af44-247ad72bd39a"). InnerVolumeSpecName "kube-api-access-vlgbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:35.958572 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.958530 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:35.958572 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.958564 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11127b7-5959-444e-af44-247ad72bd39a-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:35.958572 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:35.958574 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgbn\" (UniqueName: \"kubernetes.io/projected/a11127b7-5959-444e-af44-247ad72bd39a-kube-api-access-vlgbn\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:36.818592 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:36.818561 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz" Sep 06 14:26:36.842510 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:36.842476 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz"] Sep 06 14:26:36.845067 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:36.845041 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c443d8-6b9fb46b75-ks2vz"] Sep 06 14:26:37.236708 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:37.236671 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11127b7-5959-444e-af44-247ad72bd39a" path="/var/lib/kubelet/pods/a11127b7-5959-444e-af44-247ad72bd39a/volumes" Sep 06 14:26:44.396341 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.396292 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c955d44d-xjwbw"] Sep 06 14:26:44.396952 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.396574 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68c955d44d-xjwbw" podUID="169e3b99-bc43-4024-8970-936fb971436b" containerName="manager" containerID="cri-o://12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d" gracePeriod=30 Sep 06 14:26:44.656708 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.656630 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:26:44.737418 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.737365 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-metrics-certs\") pod \"169e3b99-bc43-4024-8970-936fb971436b\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " Sep 06 14:26:44.737627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.737520 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-webhook-cert\") pod \"169e3b99-bc43-4024-8970-936fb971436b\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " Sep 06 14:26:44.737627 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.737609 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/169e3b99-bc43-4024-8970-936fb971436b-kube-api-access-gskkj\") pod \"169e3b99-bc43-4024-8970-936fb971436b\" (UID: \"169e3b99-bc43-4024-8970-936fb971436b\") " Sep 06 14:26:44.739682 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.739641 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "169e3b99-bc43-4024-8970-936fb971436b" (UID: "169e3b99-bc43-4024-8970-936fb971436b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:44.739682 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.739674 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "169e3b99-bc43-4024-8970-936fb971436b" (UID: "169e3b99-bc43-4024-8970-936fb971436b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:26:44.739857 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.739692 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169e3b99-bc43-4024-8970-936fb971436b-kube-api-access-gskkj" (OuterVolumeSpecName: "kube-api-access-gskkj") pod "169e3b99-bc43-4024-8970-936fb971436b" (UID: "169e3b99-bc43-4024-8970-936fb971436b"). InnerVolumeSpecName "kube-api-access-gskkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:26:44.839016 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.838979 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:44.839016 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.839014 2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/169e3b99-bc43-4024-8970-936fb971436b-webhook-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:44.839251 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.839031 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/169e3b99-bc43-4024-8970-936fb971436b-kube-api-access-gskkj\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:26:44.868778 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.868741 2550 generic.go:356] "Generic (PLEG): container finished" podID="169e3b99-bc43-4024-8970-936fb971436b" containerID="12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d" exitCode=0 Sep 06 14:26:44.869028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.868791 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-xjwbw" Sep 06 14:26:44.869028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.868829 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-xjwbw" event={"ID":"169e3b99-bc43-4024-8970-936fb971436b","Type":"ContainerDied","Data":"12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d"} Sep 06 14:26:44.869028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.868906 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-xjwbw" event={"ID":"169e3b99-bc43-4024-8970-936fb971436b","Type":"ContainerDied","Data":"a4ebd412bc1f263a3098b4a575dc1223cfcf2b27dd1c1f6a732f0298c0da5c8d"} Sep 06 14:26:44.869028 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.868938 2550 scope.go:117] "RemoveContainer" containerID="12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d" Sep 06 14:26:44.881184 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.881165 2550 scope.go:117] "RemoveContainer" containerID="12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d" Sep 06 14:26:44.881539 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:26:44.881518 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d\": container with ID starting with 12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d not found: ID does not exist" containerID="12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d" Sep 06 14:26:44.881584 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.881549 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d"} err="failed to get container status \"12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d\": rpc error: code = NotFound desc = could not find container \"12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d\": container with ID starting with 12ba85e2e865a894b4d0a19b606aae10418b4c5293837023cd1bb3be9706bd9d not found: ID does not exist" Sep 06 14:26:44.890377 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.890342 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c955d44d-xjwbw"] Sep 06 14:26:44.894226 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:44.894195 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c955d44d-xjwbw"] Sep 06 14:26:45.236767 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:26:45.236727 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169e3b99-bc43-4024-8970-936fb971436b" path="/var/lib/kubelet/pods/169e3b99-bc43-4024-8970-936fb971436b/volumes" Sep 06 14:27:31.387449 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:31.387418 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:27:31.391926 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:31.391868 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:27:45.665907 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.665857 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c955d44d-tb94x"] Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666393 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8493ae2c-7d48-4ed2-89ee-6e355727daf7" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666410 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8493ae2c-7d48-4ed2-89ee-6e355727daf7" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666423 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff39c708-38d5-4084-aa9f-dfa8d3241cdd" containerName="payload-processing" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666429 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff39c708-38d5-4084-aa9f-dfa8d3241cdd" containerName="payload-processing" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666441 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169e3b99-bc43-4024-8970-936fb971436b" containerName="manager" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666447 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="169e3b99-bc43-4024-8970-936fb971436b" containerName="manager" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666455 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666460 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666476 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11127b7-5959-444e-af44-247ad72bd39a" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666481 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11127b7-5959-444e-af44-247ad72bd39a" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666490 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d341388-a8a3-4b1e-bf26-49be8f196d89" containerName="maas-api" Sep 06 14:27:45.666503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666495 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d341388-a8a3-4b1e-bf26-49be8f196d89" containerName="maas-api" Sep 06 14:27:45.666881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666568 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8493ae2c-7d48-4ed2-89ee-6e355727daf7" containerName="maas-api" Sep 06 14:27:45.666881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666581 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff39c708-38d5-4084-aa9f-dfa8d3241cdd" containerName="payload-processing" Sep 06 14:27:45.666881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666587 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d341388-a8a3-4b1e-bf26-49be8f196d89" containerName="maas-api" Sep 06 14:27:45.666881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666593 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="169e3b99-bc43-4024-8970-936fb971436b" containerName="manager" Sep 06 14:27:45.666881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666599 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11127b7-5959-444e-af44-247ad72bd39a" containerName="maas-api" Sep 06 14:27:45.666881 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.666604 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="63fe9f6d-2c01-44bb-9ba7-32f2ca10603c" containerName="maas-api" Sep 06 14:27:45.669732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.669714 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.671748 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.671725 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 14:27:45.672728 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.672710 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 14:27:45.672818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.672797 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d2fsk\"" Sep 06 14:27:45.672818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.672797 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 14:27:45.679563 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.679529 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c955d44d-tb94x"] Sep 06 14:27:45.795867 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.795825 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-webhook-cert\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.796088 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.795919 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-metrics-certs\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.796088 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.795945 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qkr\" (UniqueName: \"kubernetes.io/projected/bbc75baa-7a20-4333-9b38-d4eea12bc883-kube-api-access-j6qkr\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.896773 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.896730 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-webhook-cert\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.896985 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.896797 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-metrics-certs\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.896985 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.896840 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qkr\" (UniqueName: \"kubernetes.io/projected/bbc75baa-7a20-4333-9b38-d4eea12bc883-kube-api-access-j6qkr\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.899515 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.899484 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-metrics-certs\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.899515 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.899511 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-webhook-cert\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.904694 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.904665 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qkr\" (UniqueName: \"kubernetes.io/projected/bbc75baa-7a20-4333-9b38-d4eea12bc883-kube-api-access-j6qkr\") pod \"maas-controller-68c955d44d-tb94x\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:45.982171 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:45.982132 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:46.259605 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:46.259570 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c955d44d-tb94x"] Sep 06 14:27:46.261282 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:27:46.261250 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc75baa_7a20_4333_9b38_d4eea12bc883.slice/crio-c4106e05142e9605af0faf66bff28ea1f74a64ddba42b65659ec8368e47636d1 WatchSource:0}: Error finding container c4106e05142e9605af0faf66bff28ea1f74a64ddba42b65659ec8368e47636d1: Status 404 returned error can't find the container with id c4106e05142e9605af0faf66bff28ea1f74a64ddba42b65659ec8368e47636d1 Sep 06 14:27:47.235619 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:47.235578 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-tb94x" event={"ID":"bbc75baa-7a20-4333-9b38-d4eea12bc883","Type":"ContainerStarted","Data":"6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96"} Sep 06 14:27:47.236080 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:47.235634 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-tb94x" event={"ID":"bbc75baa-7a20-4333-9b38-d4eea12bc883","Type":"ContainerStarted","Data":"c4106e05142e9605af0faf66bff28ea1f74a64ddba42b65659ec8368e47636d1"} Sep 06 14:27:47.236080 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:47.235674 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:27:47.251963 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:47.251910 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c955d44d-tb94x" podStartSLOduration=1.8187382090000002 podStartE2EDuration="2.2518781s" totalImagesPullingTime="433.139891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:27:47.249746089 +0000 UTC m=+1816.513190813" watchObservedRunningTime="2026-09-06 14:27:47.2518781 +0000 UTC m=+1816.515322878" Sep 06 14:27:58.250768 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:27:58.250735 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:28:08.019759 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.019713 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c955d44d-tb94x"] Sep 06 14:28:08.020406 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.019974 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68c955d44d-tb94x" podUID="bbc75baa-7a20-4333-9b38-d4eea12bc883" containerName="manager" containerID="cri-o://6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96" gracePeriod=30 Sep 06 14:28:08.287011 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.286937 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:28:08.367071 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.367035 2550 generic.go:356] "Generic (PLEG): container finished" podID="bbc75baa-7a20-4333-9b38-d4eea12bc883" containerID="6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96" exitCode=0 Sep 06 14:28:08.367246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.367094 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-tb94x" Sep 06 14:28:08.367246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.367105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-tb94x" event={"ID":"bbc75baa-7a20-4333-9b38-d4eea12bc883","Type":"ContainerDied","Data":"6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96"} Sep 06 14:28:08.367246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.367149 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-tb94x" event={"ID":"bbc75baa-7a20-4333-9b38-d4eea12bc883","Type":"ContainerDied","Data":"c4106e05142e9605af0faf66bff28ea1f74a64ddba42b65659ec8368e47636d1"} Sep 06 14:28:08.367246 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.367172 2550 scope.go:117] "RemoveContainer" containerID="6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96" Sep 06 14:28:08.378886 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.378863 2550 scope.go:117] "RemoveContainer" containerID="6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96" Sep 06 14:28:08.379404 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:28:08.379375 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96\": container with ID starting with 6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96 not found: ID does not exist" containerID="6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96" Sep 06 14:28:08.379466 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.379416 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96"} err="failed to get container status \"6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96\": rpc error: code = NotFound desc = could not find container \"6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96\": container with ID starting with 6da50f57cf65b7290ec0a8f9b72f005bc2473e204fa555360f8fb0b8c7622d96 not found: ID does not exist" Sep 06 14:28:08.403552 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.403518 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-metrics-certs\") pod \"bbc75baa-7a20-4333-9b38-d4eea12bc883\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " Sep 06 14:28:08.403735 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.403603 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6qkr\" (UniqueName: \"kubernetes.io/projected/bbc75baa-7a20-4333-9b38-d4eea12bc883-kube-api-access-j6qkr\") pod \"bbc75baa-7a20-4333-9b38-d4eea12bc883\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " Sep 06 14:28:08.403735 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.403654 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-webhook-cert\") pod \"bbc75baa-7a20-4333-9b38-d4eea12bc883\" (UID: \"bbc75baa-7a20-4333-9b38-d4eea12bc883\") " Sep 06 14:28:08.405818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.405777 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc75baa-7a20-4333-9b38-d4eea12bc883-kube-api-access-j6qkr" (OuterVolumeSpecName: "kube-api-access-j6qkr") pod "bbc75baa-7a20-4333-9b38-d4eea12bc883" (UID: "bbc75baa-7a20-4333-9b38-d4eea12bc883"). InnerVolumeSpecName "kube-api-access-j6qkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:28:08.405818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.405793 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbc75baa-7a20-4333-9b38-d4eea12bc883" (UID: "bbc75baa-7a20-4333-9b38-d4eea12bc883"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:28:08.405818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.405824 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bbc75baa-7a20-4333-9b38-d4eea12bc883" (UID: "bbc75baa-7a20-4333-9b38-d4eea12bc883"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:28:08.504464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.504419 2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-webhook-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:28:08.504464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.504457 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc75baa-7a20-4333-9b38-d4eea12bc883-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:28:08.504464 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.504467 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qkr\" (UniqueName: \"kubernetes.io/projected/bbc75baa-7a20-4333-9b38-d4eea12bc883-kube-api-access-j6qkr\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:28:08.689455 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.689409 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c955d44d-tb94x"] Sep 06 14:28:08.694661 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:08.694626 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c955d44d-tb94x"] Sep 06 14:28:09.236220 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.236183 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc75baa-7a20-4333-9b38-d4eea12bc883" path="/var/lib/kubelet/pods/bbc75baa-7a20-4333-9b38-d4eea12bc883/volumes" Sep 06 14:28:09.249166 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.249113 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-68c955d44d-tb94x" podUID="bbc75baa-7a20-4333-9b38-d4eea12bc883" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.89:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 14:28:09.883918 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.883862 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c955d44d-55hkw"] Sep 06 14:28:09.884474 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.884451 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc75baa-7a20-4333-9b38-d4eea12bc883" containerName="manager" Sep 06 14:28:09.884474 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.884476 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc75baa-7a20-4333-9b38-d4eea12bc883" containerName="manager" Sep 06 14:28:09.884594 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.884569 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc75baa-7a20-4333-9b38-d4eea12bc883" containerName="manager" Sep 06 14:28:09.887660 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.887643 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:09.889689 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.889663 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 06 14:28:09.890593 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.890569 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 06 14:28:09.890728 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.890636 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d2fsk\"" Sep 06 14:28:09.890728 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.890645 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 06 14:28:09.897317 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.897288 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c955d44d-55hkw"] Sep 06 14:28:09.915422 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.915368 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbx2\" (UniqueName: \"kubernetes.io/projected/514ad906-5168-462b-8804-dd52fda7edee-kube-api-access-qrbx2\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:09.915611 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.915435 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ad906-5168-462b-8804-dd52fda7edee-metrics-certs\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:09.915611 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:09.915488 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/514ad906-5168-462b-8804-dd52fda7edee-webhook-cert\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.016446 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.016405 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/514ad906-5168-462b-8804-dd52fda7edee-webhook-cert\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.016668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.016482 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbx2\" (UniqueName: \"kubernetes.io/projected/514ad906-5168-462b-8804-dd52fda7edee-kube-api-access-qrbx2\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.016668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.016516 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ad906-5168-462b-8804-dd52fda7edee-metrics-certs\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.019225 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.019193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/514ad906-5168-462b-8804-dd52fda7edee-webhook-cert\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.019347 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.019193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ad906-5168-462b-8804-dd52fda7edee-metrics-certs\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.023781 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.023750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbx2\" (UniqueName: \"kubernetes.io/projected/514ad906-5168-462b-8804-dd52fda7edee-kube-api-access-qrbx2\") pod \"maas-controller-68c955d44d-55hkw\" (UID: \"514ad906-5168-462b-8804-dd52fda7edee\") " pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.200204 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.200150 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:10.478465 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:10.478440 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c955d44d-55hkw"] Sep 06 14:28:10.480021 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:28:10.479995 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514ad906_5168_462b_8804_dd52fda7edee.slice/crio-e655993afac6f89cffe57076080032115dccef3aae6b72c4093dda06eecd6fde WatchSource:0}: Error finding container e655993afac6f89cffe57076080032115dccef3aae6b72c4093dda06eecd6fde: Status 404 returned error can't find the container with id e655993afac6f89cffe57076080032115dccef3aae6b72c4093dda06eecd6fde Sep 06 14:28:11.387867 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:11.387828 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-55hkw" event={"ID":"514ad906-5168-462b-8804-dd52fda7edee","Type":"ContainerStarted","Data":"1cbba8409a2936c31bafd63410d2f9a99138a47775c95e7b690e847ae8f4f12b"} Sep 06 14:28:11.388048 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:11.387909 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c955d44d-55hkw" event={"ID":"514ad906-5168-462b-8804-dd52fda7edee","Type":"ContainerStarted","Data":"e655993afac6f89cffe57076080032115dccef3aae6b72c4093dda06eecd6fde"} Sep 06 14:28:11.388125 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:11.388062 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:28:11.402428 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:11.402374 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c955d44d-55hkw" podStartSLOduration=2.041693919 podStartE2EDuration="2.402353286s" totalImagesPullingTime="360.659367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:28:11.40163856 +0000 UTC m=+1840.665083284" watchObservedRunningTime="2026-09-06 14:28:11.402353286 +0000 UTC m=+1840.665798012" Sep 06 14:28:22.395767 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:28:22.395735 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c955d44d-55hkw" Sep 06 14:29:17.113847 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:17.113805 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp"] Sep 06 14:29:17.114409 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:17.114145 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" podUID="39609391-2670-4881-86db-a2988228752b" containerName="manager" containerID="cri-o://6d4c6e2b6ce34a391c857a83e1417e17ddde6fe253bdd8f9ff6e4f1447369533" gracePeriod=10 Sep 06 14:29:17.792811 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:17.792722 2550 generic.go:356] "Generic (PLEG): container finished" podID="39609391-2670-4881-86db-a2988228752b" containerID="6d4c6e2b6ce34a391c857a83e1417e17ddde6fe253bdd8f9ff6e4f1447369533" exitCode=0 Sep 06 14:29:17.792811 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:17.792794 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" event={"ID":"39609391-2670-4881-86db-a2988228752b","Type":"ContainerDied","Data":"6d4c6e2b6ce34a391c857a83e1417e17ddde6fe253bdd8f9ff6e4f1447369533"} Sep 06 14:29:18.179302 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.179272 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:29:18.212862 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.212810 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdls9\" (UniqueName: \"kubernetes.io/projected/39609391-2670-4881-86db-a2988228752b-kube-api-access-pdls9\") pod \"39609391-2670-4881-86db-a2988228752b\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " Sep 06 14:29:18.213084 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.212966 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39609391-2670-4881-86db-a2988228752b-extensions-socket-volume\") pod \"39609391-2670-4881-86db-a2988228752b\" (UID: \"39609391-2670-4881-86db-a2988228752b\") " Sep 06 14:29:18.213359 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.213329 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39609391-2670-4881-86db-a2988228752b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "39609391-2670-4881-86db-a2988228752b" (UID: "39609391-2670-4881-86db-a2988228752b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 14:29:18.215020 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.214995 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39609391-2670-4881-86db-a2988228752b-kube-api-access-pdls9" (OuterVolumeSpecName: "kube-api-access-pdls9") pod "39609391-2670-4881-86db-a2988228752b" (UID: "39609391-2670-4881-86db-a2988228752b"). InnerVolumeSpecName "kube-api-access-pdls9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:29:18.313929 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.313855 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdls9\" (UniqueName: \"kubernetes.io/projected/39609391-2670-4881-86db-a2988228752b-kube-api-access-pdls9\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:29:18.313929 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.313921 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39609391-2670-4881-86db-a2988228752b-extensions-socket-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:29:18.800443 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.800414 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" Sep 06 14:29:18.800659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.800461 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp" event={"ID":"39609391-2670-4881-86db-a2988228752b","Type":"ContainerDied","Data":"b76d4b84ae1843f767f8ab6d554c88e60512f68558cdb084a287c1134b3daeae"} Sep 06 14:29:18.800659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.800529 2550 scope.go:117] "RemoveContainer" containerID="6d4c6e2b6ce34a391c857a83e1417e17ddde6fe253bdd8f9ff6e4f1447369533" Sep 06 14:29:18.821702 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.821670 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp"] Sep 06 14:29:18.825230 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:18.825201 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-nmnnp"] Sep 06 14:29:19.237205 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:29:19.237170 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39609391-2670-4881-86db-a2988228752b" path="/var/lib/kubelet/pods/39609391-2670-4881-86db-a2988228752b/volumes" Sep 06 14:30:17.819405 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.819354 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t"] Sep 06 14:30:17.820029 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.819967 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39609391-2670-4881-86db-a2988228752b" containerName="manager" Sep 06 14:30:17.820029 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.819987 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="39609391-2670-4881-86db-a2988228752b" containerName="manager" Sep 06 14:30:17.820163 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.820110 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="39609391-2670-4881-86db-a2988228752b" containerName="manager" Sep 06 14:30:17.823290 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.823270 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:17.826818 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.826795 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dchdk\"" Sep 06 14:30:17.841143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.841107 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t"] Sep 06 14:30:17.853471 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.853426 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbl6\" (UniqueName: \"kubernetes.io/projected/9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6-kube-api-access-zqbl6\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w264t\" (UID: \"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:17.853669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.853568 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w264t\" (UID: \"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:17.954189 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.954147 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w264t\" (UID: \"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:17.954382 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.954264 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbl6\" (UniqueName: \"kubernetes.io/projected/9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6-kube-api-access-zqbl6\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w264t\" (UID: \"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:17.954565 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.954545 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w264t\" (UID: \"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:17.962579 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:17.962548 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbl6\" (UniqueName: \"kubernetes.io/projected/9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6-kube-api-access-zqbl6\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w264t\" (UID: \"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:18.134358 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:18.134262 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:18.419235 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:18.419206 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t"] Sep 06 14:30:18.419837 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:30:18.419807 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9dc342_5bdf_4c62_9dac_6f0f2ff736c6.slice/crio-170d2bcb700d031824e5a35baef922276d1481cccbdfebd6f275bd937f57e085 WatchSource:0}: Error finding container 170d2bcb700d031824e5a35baef922276d1481cccbdfebd6f275bd937f57e085: Status 404 returned error can't find the container with id 170d2bcb700d031824e5a35baef922276d1481cccbdfebd6f275bd937f57e085 Sep 06 14:30:19.169111 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:19.169079 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" event={"ID":"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6","Type":"ContainerStarted","Data":"d8eec873c665a24b8dc1135546cd30363db5b5bef95b31f5656c92e1974e4887"} Sep 06 14:30:19.169507 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:19.169139 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:30:19.169507 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:19.169156 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" event={"ID":"9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6","Type":"ContainerStarted","Data":"170d2bcb700d031824e5a35baef922276d1481cccbdfebd6f275bd937f57e085"} Sep 06 14:30:19.189861 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:19.189809 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" podStartSLOduration=2.189792298 podStartE2EDuration="2.189792298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:30:19.187819743 +0000 UTC m=+1968.451264467" watchObservedRunningTime="2026-09-06 14:30:19.189792298 +0000 UTC m=+1968.453237022" Sep 06 14:30:30.177698 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:30:30.177665 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w264t" Sep 06 14:32:31.435971 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:31.435941 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:32:31.440469 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:31.440438 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:32:45.298378 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.298341 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6669d75c8f-k948n"] Sep 06 14:32:45.302440 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.302413 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.311013 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.310979 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669d75c8f-k948n"] Sep 06 14:32:45.400939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.400873 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7a25c-e41d-408e-908e-b8e0eab4b838-maas-api-tls\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.400939 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.400952 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msp2r\" (UniqueName: \"kubernetes.io/projected/70b7a25c-e41d-408e-908e-b8e0eab4b838-kube-api-access-msp2r\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.401170 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.401033 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7a25c-e41d-408e-908e-b8e0eab4b838-openshift-service-ca\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.401170 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.401063 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7a25c-e41d-408e-908e-b8e0eab4b838-metrics-certs\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.501845 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.501804 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7a25c-e41d-408e-908e-b8e0eab4b838-maas-api-tls\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.502077 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.501865 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msp2r\" (UniqueName: \"kubernetes.io/projected/70b7a25c-e41d-408e-908e-b8e0eab4b838-kube-api-access-msp2r\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.502077 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.501975 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7a25c-e41d-408e-908e-b8e0eab4b838-openshift-service-ca\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.502077 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.502000 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7a25c-e41d-408e-908e-b8e0eab4b838-metrics-certs\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.502696 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.502666 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7a25c-e41d-408e-908e-b8e0eab4b838-openshift-service-ca\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.504491 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.504457 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7a25c-e41d-408e-908e-b8e0eab4b838-metrics-certs\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.504607 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.504493 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7a25c-e41d-408e-908e-b8e0eab4b838-maas-api-tls\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.509131 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.509101 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msp2r\" (UniqueName: \"kubernetes.io/projected/70b7a25c-e41d-408e-908e-b8e0eab4b838-kube-api-access-msp2r\") pod \"maas-api-6669d75c8f-k948n\" (UID: \"70b7a25c-e41d-408e-908e-b8e0eab4b838\") " pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.617275 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.617178 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:45.894248 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.894073 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669d75c8f-k948n"] Sep 06 14:32:45.896877 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:32:45.896851 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b7a25c_e41d_408e_908e_b8e0eab4b838.slice/crio-965d4c6e1f07c49c8f72a990d9ff87dfbae766b689c97106c9f2c4122384c3b6 WatchSource:0}: Error finding container 965d4c6e1f07c49c8f72a990d9ff87dfbae766b689c97106c9f2c4122384c3b6: Status 404 returned error can't find the container with id 965d4c6e1f07c49c8f72a990d9ff87dfbae766b689c97106c9f2c4122384c3b6 Sep 06 14:32:45.898292 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:45.898273 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 14:32:46.039267 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:46.039234 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" event={"ID":"70b7a25c-e41d-408e-908e-b8e0eab4b838","Type":"ContainerStarted","Data":"965d4c6e1f07c49c8f72a990d9ff87dfbae766b689c97106c9f2c4122384c3b6"} Sep 06 14:32:47.047656 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:47.047621 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" event={"ID":"70b7a25c-e41d-408e-908e-b8e0eab4b838","Type":"ContainerStarted","Data":"2c55500a4539a6d9201b8332109c81d0aaba42ca7e4009d59e50605266cff8a4"} Sep 06 14:32:47.048128 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:47.047848 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:32:47.064299 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:47.064245 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" podStartSLOduration=1.675241968 podStartE2EDuration="2.064228882s" totalImagesPullingTime="388.986914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:32:47.062867418 +0000 UTC m=+2116.326312144" watchObservedRunningTime="2026-09-06 14:32:47.064228882 +0000 UTC m=+2116.327673605" Sep 06 14:32:53.059054 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:32:53.059020 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6669d75c8f-k948n" Sep 06 14:33:00.425342 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:00.425302 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf"] Sep 06 14:33:00.425989 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:00.425526 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" podUID="e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" containerName="maas-api" containerID="cri-o://d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f" gracePeriod=30 Sep 06 14:33:01.691649 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.691617 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:33:01.760324 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.760229 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-metrics-certs\") pod \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " Sep 06 14:33:01.760324 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.760298 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hnqq\" (UniqueName: \"kubernetes.io/projected/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-kube-api-access-5hnqq\") pod \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " Sep 06 14:33:01.760519 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.760332 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-maas-api-tls\") pod \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " Sep 06 14:33:01.760519 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.760398 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-openshift-service-ca\") pod \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\" (UID: \"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12\") " Sep 06 14:33:01.760922 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.760864 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" (UID: "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 14:33:01.762495 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.762464 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" (UID: "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:33:01.762495 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.762491 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-kube-api-access-5hnqq" (OuterVolumeSpecName: "kube-api-access-5hnqq") pod "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" (UID: "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12"). InnerVolumeSpecName "kube-api-access-5hnqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 14:33:01.762659 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.762564 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" (UID: "e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 14:33:01.862185 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.862145 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-metrics-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:33:01.862185 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.862179 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnqq\" (UniqueName: \"kubernetes.io/projected/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-kube-api-access-5hnqq\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:33:01.862185 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.862195 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-maas-api-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:33:01.862438 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:01.862208 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12-openshift-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\"" Sep 06 14:33:02.140336 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.140242 2550 generic.go:356] "Generic (PLEG): container finished" podID="e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" containerID="d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f" exitCode=0 Sep 06 14:33:02.140336 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.140287 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" Sep 06 14:33:02.140336 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.140286 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" event={"ID":"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12","Type":"ContainerDied","Data":"d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f"} Sep 06 14:33:02.140597 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.140353 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf" event={"ID":"e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12","Type":"ContainerDied","Data":"6bfd8f2b70c85245635c6bc3d1e12c8fc53849d8fa667624669c12a60b317666"} Sep 06 14:33:02.140597 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.140385 2550 scope.go:117] "RemoveContainer" containerID="d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f" Sep 06 14:33:02.150719 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.150701 2550 scope.go:117] "RemoveContainer" containerID="d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f" Sep 06 14:33:02.151071 ip-10-0-141-9 kubenswrapper[2550]: E0906 14:33:02.151052 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f\": container with ID starting with d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f not found: ID does not exist" containerID="d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f" Sep 06 14:33:02.151143 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.151080 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f"} err="failed to get container status \"d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f\": rpc error: code = NotFound desc = could not find container \"d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f\": container with ID starting with d091b791922fb41d274fc8d01acb21dff5ccfaf5db53eb5bd3dce916b47c482f not found: ID does not exist" Sep 06 14:33:02.161174 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.161138 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf"] Sep 06 14:33:02.166715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:02.166675 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff94db9d5-jhssf"] Sep 06 14:33:03.236759 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:03.236718 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" path="/var/lib/kubelet/pods/e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12/volumes" Sep 06 14:33:20.909840 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:20.909800 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-849585cd49-2jzbw_b17a3030-3190-4541-93f7-a3e3e7da7a60/authorino/0.log" Sep 06 14:33:25.933310 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:25.933274 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8cbqr_2ffe7fd6-4919-4657-b2e2-7aa13212f42e/manager/0.log" Sep 06 14:33:26.047089 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:26.047054 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h8sc8_3040babf-f012-4c50-9998-274aebf42331/manager/0.log" Sep 06 14:33:26.160235 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:26.160197 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c955d44d-55hkw_514ad906-5168-462b-8804-dd52fda7edee/manager/0.log" Sep 06 14:33:26.494666 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:26.494633 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-vhprj_dce402da-d573-4c7a-afb5-e79852b0dc02/manager/0.log" Sep 06 14:33:26.500098 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:26.500072 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-vhprj_dce402da-d573-4c7a-afb5-e79852b0dc02/copy-manifests/0.log" Sep 06 14:33:27.052652 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:27.052615 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6669d75c8f-k948n_70b7a25c-e41d-408e-908e-b8e0eab4b838/maas-api/0.log" Sep 06 14:33:27.466576 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:27.466540 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fhcj8_71ca09de-9023-4dab-b313-72b2387d2973/postgres/0.log" Sep 06 14:33:28.210859 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:28.210824 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-849585cd49-2jzbw_b17a3030-3190-4541-93f7-a3e3e7da7a60/authorino/0.log" Sep 06 14:33:28.327299 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:28.327268 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-d6ffc_2f91e93b-7407-430f-93aa-e15b988d83c9/manager/0.log" Sep 06 14:33:28.436124 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:28.436089 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-c5x97_5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8/manager/0.log" Sep 06 14:33:28.541398 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:28.541322 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-nnc2h_7fd340df-cffa-4e0c-acd0-4dff5d500fda/kuadrant-console-plugin/0.log" Sep 06 14:33:28.654109 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:28.654060 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-w264t_9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6/manager/0.log" Sep 06 14:33:28.883919 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:28.883825 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-2mjfg_5bcb678a-dc78-45f5-831a-8eafdd9ecaae/manager/0.log" Sep 06 14:33:29.452870 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:29.452835 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-5l4pj_8c2bdce0-1beb-425b-a752-d49b305fd470/kube-auth-proxy/0.log" Sep 06 14:33:29.780605 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:29.780521 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hlqmr_013d32af-a6e0-4dbf-8800-7685bd8335f3/payload-pre-processing/0.log" Sep 06 14:33:30.009142 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:30.009111 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6c6bb47b-xc5lw_7c679d17-50c8-4f87-b37e-fdd45dbee2bf/router/0.log" Sep 06 14:33:30.340826 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:30.340797 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-krxwd_3f30cbf8-2e79-44d3-bf50-cd46ce4cbad8/main/0.log" Sep 06 14:33:30.658051 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:30.657973 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qbkg2_fdf95a9c-ba62-40e1-befb-2867ef33decb/main/0.log" Sep 06 14:33:39.484421 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:39.484381 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b7m9q_c61bfe89-af85-4a64-a403-860883b0890b/global-pull-secret-syncer/0.log" Sep 06 14:33:39.636566 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:39.636536 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zh6zz_6ed959d9-887f-4d39-9f00-015d657dd5ae/konnectivity-agent/0.log" Sep 06 14:33:39.690360 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:39.690330 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-9.ec2.internal_4e79bb2805f50e77c363fc54fc65fe32/haproxy/0.log" Sep 06 14:33:44.461946 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:44.461907 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-849585cd49-2jzbw_b17a3030-3190-4541-93f7-a3e3e7da7a60/authorino/0.log" Sep 06 14:33:44.508640 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:44.508605 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-d6ffc_2f91e93b-7407-430f-93aa-e15b988d83c9/manager/0.log" Sep 06 14:33:44.535612 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:44.535581 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-c5x97_5493ae1f-518d-4ce1-8e4e-1aff9a0bd8b8/manager/0.log" Sep 06 14:33:44.561027 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:44.561000 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-nnc2h_7fd340df-cffa-4e0c-acd0-4dff5d500fda/kuadrant-console-plugin/0.log" Sep 06 14:33:44.607608 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:44.607571 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-w264t_9f9dc342-5bdf-4c62-9dac-6f0f2ff736c6/manager/0.log" Sep 06 14:33:44.679812 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:44.679775 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-2mjfg_5bcb678a-dc78-45f5-831a-8eafdd9ecaae/manager/0.log" Sep 06 14:33:46.221440 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:46.221409 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-gscwv_019adef7-a1fb-4778-8880-2b206ee04a44/cluster-monitoring-operator/0.log" Sep 06 14:33:46.518025 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:46.517949 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pzff7_3cc3ddb6-10e2-4ad6-9d66-90f36e734a26/node-exporter/0.log" Sep 06 14:33:46.536370 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:46.536350 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pzff7_3cc3ddb6-10e2-4ad6-9d66-90f36e734a26/kube-rbac-proxy/0.log" Sep 06 14:33:46.558942 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:46.558921 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pzff7_3cc3ddb6-10e2-4ad6-9d66-90f36e734a26/init-textfile/0.log" Sep 06 14:33:48.576520 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:48.576492 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-q6dld_93551ec1-12b7-43cf-85bd-d63119d7087a/networking-console-plugin/0.log" Sep 06 14:33:49.162836 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.162800 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/2.log" Sep 06 14:33:49.166979 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.166951 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ss4gj_0e557111-94a1-498c-a4f9-273705d374be/console-operator/3.log" Sep 06 14:33:49.684261 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.684230 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-847f87b568-vf7t2_bbf81eea-6288-4204-83b8-ea3c13b25c9f/console/0.log" Sep 06 14:33:49.771210 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.771175 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7"] Sep 06 14:33:49.771622 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.771609 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" containerName="maas-api" Sep 06 14:33:49.771669 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.771623 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" containerName="maas-api" Sep 06 14:33:49.771709 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.771696 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3859a8b-83c4-4f17-8ac9-8ca3dfb4ea12" containerName="maas-api" Sep 06 14:33:49.774936 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.774920 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.777154 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.777127 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfljt\"/\"openshift-service-ca.crt\"" Sep 06 14:33:49.777289 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.777202 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfljt\"/\"kube-root-ca.crt\"" Sep 06 14:33:49.778046 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.778028 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfljt\"/\"default-dockercfg-9w4wc\"" Sep 06 14:33:49.783391 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.783365 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7"] Sep 06 14:33:49.884828 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.884786 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-lib-modules\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.884828 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.884841 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpqx\" (UniqueName: \"kubernetes.io/projected/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-kube-api-access-vjpqx\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.885085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.884866 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-proc\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.885085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.884963 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-podres\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.885085 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.884987 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-sys\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986141 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-podres\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986188 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-sys\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986237 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-lib-modules\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986517 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpqx\" (UniqueName: \"kubernetes.io/projected/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-kube-api-access-vjpqx\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986517 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986265 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-proc\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986517 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986328 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-podres\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986517 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986345 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-sys\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986517 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986335 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-proc\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.986517 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.986383 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-lib-modules\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:49.993809 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:49.993777 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpqx\" (UniqueName: \"kubernetes.io/projected/d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8-kube-api-access-vjpqx\") pod \"perf-node-gather-daemonset-nwwz7\" (UID: \"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:50.085617 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:50.085568 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:50.360876 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:50.360846 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7"] Sep 06 14:33:50.362455 ip-10-0-141-9 kubenswrapper[2550]: W0906 14:33:50.362428 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd07129f2_bdf0_4e86_a1ee_ef6ae9ab47d8.slice/crio-ed9961f6ab587c53d5d21ca61ac9d1af7c8705b2d34dd69fab4a8f1f5cc4cc04 WatchSource:0}: Error finding container ed9961f6ab587c53d5d21ca61ac9d1af7c8705b2d34dd69fab4a8f1f5cc4cc04: Status 404 returned error can't find the container with id ed9961f6ab587c53d5d21ca61ac9d1af7c8705b2d34dd69fab4a8f1f5cc4cc04 Sep 06 14:33:50.413849 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:50.413822 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-b6w52_46f7ff31-5a17-4585-b2f9-a2d4981f00d4/volume-data-source-validator/0.log" Sep 06 14:33:50.439732 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:50.439694 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" event={"ID":"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8","Type":"ContainerStarted","Data":"ed9961f6ab587c53d5d21ca61ac9d1af7c8705b2d34dd69fab4a8f1f5cc4cc04"} Sep 06 14:33:51.446839 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:51.446798 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" event={"ID":"d07129f2-bdf0-4e86-a1ee-ef6ae9ab47d8","Type":"ContainerStarted","Data":"ccf7ea268c63470166f77d8c822710323c58a4d9922bc10fafc72fab75cf513d"} Sep 06 14:33:51.447495 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:51.446941 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:51.461584 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:51.461528 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" podStartSLOduration=2.461510749 podStartE2EDuration="2.461510749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 14:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:33:51.459679383 +0000 UTC m=+2180.723124107" watchObservedRunningTime="2026-09-06 14:33:51.461510749 +0000 UTC m=+2180.724955473" Sep 06 14:33:51.647841 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:51.647811 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pd4sj_0b13bd22-de6a-4432-92d0-8990aee7c811/dns/0.log" Sep 06 14:33:51.665646 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:51.665617 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pd4sj_0b13bd22-de6a-4432-92d0-8990aee7c811/kube-rbac-proxy/0.log" Sep 06 14:33:51.705164 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:51.705086 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hv2l_1a1e2bb6-a11c-4520-af37-b832772ecd1b/dns-node-resolver/0.log" Sep 06 14:33:52.314820 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:52.314791 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nq79q_94411733-7637-47ae-a05c-30a723c94748/node-ca/0.log" Sep 06 14:33:53.755758 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:53.755728 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-5l4pj_8c2bdce0-1beb-425b-a752-d49b305fd470/kube-auth-proxy/0.log" Sep 06 14:33:53.852374 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:53.852330 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hlqmr_013d32af-a6e0-4dbf-8800-7685bd8335f3/payload-pre-processing/0.log" Sep 06 14:33:53.919509 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:53.919471 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6c6bb47b-xc5lw_7c679d17-50c8-4f87-b37e-fdd45dbee2bf/router/0.log" Sep 06 14:33:54.806546 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:54.806514 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6hq6_ce47be1d-02c9-4e8b-866b-2531c424bc2c/serve-healthcheck-canary/0.log" Sep 06 14:33:55.237233 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:55.237204 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vr22w_b1209f84-910d-4d92-ae80-c5eaac7cdb1f/insights-operator/0.log" Sep 06 14:33:55.238917 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:55.238881 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vr22w_b1209f84-910d-4d92-ae80-c5eaac7cdb1f/insights-operator/1.log" Sep 06 14:33:55.381248 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:55.381221 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kvrcs_afd0bcff-9417-4f98-af61-50c78ce4f089/kube-rbac-proxy/0.log" Sep 06 14:33:55.398691 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:55.398669 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kvrcs_afd0bcff-9417-4f98-af61-50c78ce4f089/exporter/0.log" Sep 06 14:33:55.418503 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:55.418480 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kvrcs_afd0bcff-9417-4f98-af61-50c78ce4f089/extractor/0.log" Sep 06 14:33:57.456006 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:57.455973 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8cbqr_2ffe7fd6-4919-4657-b2e2-7aa13212f42e/manager/0.log" Sep 06 14:33:57.464231 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:57.464208 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-nwwz7" Sep 06 14:33:57.638715 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:57.638684 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h8sc8_3040babf-f012-4c50-9998-274aebf42331/manager/0.log" Sep 06 14:33:57.687602 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:57.687562 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c955d44d-55hkw_514ad906-5168-462b-8804-dd52fda7edee/manager/0.log" Sep 06 14:33:57.845343 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:57.845265 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-vhprj_dce402da-d573-4c7a-afb5-e79852b0dc02/manager/0.log" Sep 06 14:33:57.864641 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:57.864614 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-vhprj_dce402da-d573-4c7a-afb5-e79852b0dc02/copy-manifests/0.log" Sep 06 14:33:59.508171 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:33:59.508099 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-b6g5l_84c55162-9491-42d8-8a62-9a47673c3c5c/openshift-lws-operator/0.log" Sep 06 14:34:00.708410 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:00.708378 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-5l4pj_8c2bdce0-1beb-425b-a752-d49b305fd470/kube-auth-proxy/0.log" Sep 06 14:34:00.782784 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:00.782748 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hlqmr_013d32af-a6e0-4dbf-8800-7685bd8335f3/payload-pre-processing/0.log" Sep 06 14:34:00.835170 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:00.835129 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6c6bb47b-xc5lw_7c679d17-50c8-4f87-b37e-fdd45dbee2bf/router/0.log" Sep 06 14:34:02.937331 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:02.937299 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-5l4pj_8c2bdce0-1beb-425b-a752-d49b305fd470/kube-auth-proxy/0.log" Sep 06 14:34:03.013717 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:03.013674 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hlqmr_013d32af-a6e0-4dbf-8800-7685bd8335f3/payload-pre-processing/0.log" Sep 06 14:34:03.071243 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:03.071210 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6c6bb47b-xc5lw_7c679d17-50c8-4f87-b37e-fdd45dbee2bf/router/0.log" Sep 06 14:34:04.512019 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:04.511989 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5wbh8_379fa1ea-74c0-4b8c-97a9-279f8d70f952/migrator/0.log" Sep 06 14:34:04.532094 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:04.532065 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5wbh8_379fa1ea-74c0-4b8c-97a9-279f8d70f952/graceful-termination/0.log" Sep 06 14:34:04.876787 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:04.876707 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-7qqkq_7be59bd6-458a-47f0-9e1f-4e22c97a8553/kube-storage-version-migrator-operator/1.log" Sep 06 14:34:04.877668 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:04.877646 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-7qqkq_7be59bd6-458a-47f0-9e1f-4e22c97a8553/kube-storage-version-migrator-operator/0.log" Sep 06 14:34:06.030163 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.030135 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/kube-multus-additional-cni-plugins/0.log" Sep 06 14:34:06.048110 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.047994 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/egress-router-binary-copy/0.log" Sep 06 14:34:06.065161 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.065123 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/cni-plugins/0.log" Sep 06 14:34:06.081396 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.081367 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/bond-cni-plugin/0.log" Sep 06 14:34:06.098615 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.098563 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/routeoverride-cni/0.log" Sep 06 14:34:06.117938 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.117861 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/whereabouts-cni-bincopy/0.log" Sep 06 14:34:06.134754 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.134731 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzz6f_07010b81-f41e-4dd2-aced-6ce550d937ce/whereabouts-cni/0.log" Sep 06 14:34:06.274735 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.274695 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xkbv8_b3ff0bbe-649f-4370-9877-d4fe5b184c99/kube-multus/0.log" Sep 06 14:34:06.370254 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.370171 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tw57c_7025a487-10f2-475a-8da2-30f50c9989c0/network-metrics-daemon/0.log" Sep 06 14:34:06.385319 ip-10-0-141-9 kubenswrapper[2550]: I0906 14:34:06.385292 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tw57c_7025a487-10f2-475a-8da2-30f50c9989c0/kube-rbac-proxy/0.log"