Sep 19 12:25:02.957040 ip-10-0-131-149 systemd[1]: Starting Kubernetes Kubelet... Sep 19 12:25:06.331219 ip-10-0-131-149 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 12:25:06.331219 ip-10-0-131-149 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 19 12:25:06.331219 ip-10-0-131-149 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 12:25:06.331219 ip-10-0-131-149 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 19 12:25:06.331219 ip-10-0-131-149 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 12:25:06.331983 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.331311 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 19 12:25:06.334329 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334312 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 12:25:06.334329 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334329 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334334 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334338 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334341 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334344 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334347 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334350 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334352 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334355 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334358 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334360 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334363 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334365 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334368 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334372 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334375 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334378 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334381 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334383 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334385 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 12:25:06.334394 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334389 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334391 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334394 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334397 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334399 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334402 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334405 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334408 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334411 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334413 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334416 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334418 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334421 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334423 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334426 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334428 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334431 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334433 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334435 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334438 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 12:25:06.334945 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334440 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334442 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334445 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334447 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334450 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334452 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334454 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334457 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334459 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334462 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334464 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334467 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334470 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334473 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334479 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334484 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334487 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334490 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334493 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 12:25:06.335763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334496 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334499 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334502 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334505 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334507 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334510 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334513 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334516 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334519 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334522 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334524 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334527 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334530 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334533 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334536 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334540 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334543 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334545 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334547 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334550 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 12:25:06.336261 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334553 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334555 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334558 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334560 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334563 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334565 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334568 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334572 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334575 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334577 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334580 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334582 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334584 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334587 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334589 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334592 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334595 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334598 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.334600 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336620 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 12:25:06.336775 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336632 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336637 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336640 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336644 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336647 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336650 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336653 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336655 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336658 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336660 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336663 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336666 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336669 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336671 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336674 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336677 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336679 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336699 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336702 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336705 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 12:25:06.337258 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336708 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336711 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336714 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336717 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336719 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336723 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336725 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336728 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336731 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336733 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336736 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336739 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336743 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336746 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336748 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336751 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336754 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336756 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336759 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 12:25:06.337774 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336762 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336764 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336767 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336769 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336771 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336774 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336776 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336779 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336782 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336784 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336788 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336790 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336793 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336795 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336797 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336800 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336802 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336805 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336808 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336810 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 12:25:06.338250 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336813 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336816 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336819 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336822 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336824 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336827 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336830 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336833 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336835 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336838 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336841 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336843 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336846 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336851 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336855 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336858 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336861 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336864 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336867 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 12:25:06.338757 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336871 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336874 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336877 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336880 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336883 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336886 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336888 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336891 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336893 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336896 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336899 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336901 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336904 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336906 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336909 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336912 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336914 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336918 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336922 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.336925 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 12:25:06.339224 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337316 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337321 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337324 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337327 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337330 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337333 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337335 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337338 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337341 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337344 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337346 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337349 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337352 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337354 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337357 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337360 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337363 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337366 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337368 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337370 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 12:25:06.339729 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337373 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337376 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337380 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337382 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337385 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337388 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337390 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337393 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337395 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337398 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337400 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337403 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337408 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337412 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337415 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337418 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337422 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337425 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337428 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 12:25:06.340207 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337430 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337433 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337436 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337438 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337441 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337443 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337446 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337448 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337451 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337453 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337456 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337459 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337461 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337476 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337479 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337483 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337485 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337488 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337490 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 12:25:06.340724 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337493 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337496 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337498 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337501 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337504 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337508 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337510 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337513 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337515 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337518 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337521 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337523 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337526 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337528 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337531 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337533 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337535 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337538 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337540 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337543 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 12:25:06.341186 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337546 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337548 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337551 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337553 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337557 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337559 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337562 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337565 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337567 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337570 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337573 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337576 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337578 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337581 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337583 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337585 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337590 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337593 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337595 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337598 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 12:25:06.341691 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.337600 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337677 2585 flags.go:64] FLAG: --address="0.0.0.0" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337730 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337737 2585 flags.go:64] FLAG: --anonymous-auth="true" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337742 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337746 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337749 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337753 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337758 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337761 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337764 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337768 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337771 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337774 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337777 2585 flags.go:64] FLAG: --cgroup-root="" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337780 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337783 2585 flags.go:64] FLAG: --client-ca-file="" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337787 2585 flags.go:64] FLAG: --cloud-provider="external" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337790 2585 flags.go:64] FLAG: --cluster-dns="[]" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337795 2585 flags.go:64] FLAG: --cluster-domain="" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337798 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337801 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337805 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337808 2585 flags.go:64] FLAG: --container-log-max-files="5" Sep 19 12:25:06.342203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337812 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337815 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337818 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337822 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337827 2585 flags.go:64] FLAG: --contention-profiling="false" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337830 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337833 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337836 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337839 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337843 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337846 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337849 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337852 2585 flags.go:64] FLAG: --enable-load-reader="false" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337855 2585 flags.go:64] FLAG: --enable-server="true" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337858 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337863 2585 flags.go:64] FLAG: --event-burst="100" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337866 2585 flags.go:64] FLAG: --event-qps="50" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337869 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337871 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337875 2585 flags.go:64] FLAG: --eviction-hard="" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337879 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337882 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337884 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337887 2585 flags.go:64] FLAG: --eviction-soft="" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337890 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 19 12:25:06.342793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337893 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337897 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337900 2585 flags.go:64] FLAG: --experimental-mounter-path="" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337903 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337906 2585 flags.go:64] FLAG: --fail-swap-on="true" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337909 2585 flags.go:64] FLAG: --feature-gates="" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337912 2585 flags.go:64] FLAG: --file-check-frequency="20s" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337916 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337919 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337921 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337925 2585 flags.go:64] FLAG: --healthz-port="10248" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337929 2585 flags.go:64] FLAG: --help="false" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337932 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337935 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337938 2585 flags.go:64] FLAG: --http-check-frequency="20s" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337941 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337945 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337948 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337951 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337954 2585 flags.go:64] FLAG: --image-service-endpoint="" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337957 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337959 2585 flags.go:64] FLAG: --kube-api-burst="100" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337962 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337965 2585 flags.go:64] FLAG: --kube-api-qps="50" Sep 19 12:25:06.343426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337968 2585 flags.go:64] FLAG: --kube-reserved="" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337971 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337974 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337977 2585 flags.go:64] FLAG: --kubelet-cgroups="" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337979 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337982 2585 flags.go:64] FLAG: --lock-file="" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337985 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337988 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337990 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.337996 2585 flags.go:64] FLAG: --log-json-split-stream="false" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338000 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338003 2585 flags.go:64] FLAG: --log-text-split-stream="false" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338006 2585 flags.go:64] FLAG: --logging-format="text" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338008 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338012 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338015 2585 flags.go:64] FLAG: --manifest-url="" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338017 2585 flags.go:64] FLAG: --manifest-url-header="" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338022 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338024 2585 flags.go:64] FLAG: --max-open-files="1000000" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338030 2585 flags.go:64] FLAG: --max-pods="110" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338033 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338036 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338039 2585 flags.go:64] FLAG: --memory-manager-policy="None" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338041 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338044 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 19 12:25:06.344090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338047 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338050 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338058 2585 flags.go:64] FLAG: --node-status-max-images="50" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338062 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338064 2585 flags.go:64] FLAG: --oom-score-adj="-999" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338067 2585 flags.go:64] FLAG: --pod-cidr="" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338070 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338076 2585 flags.go:64] FLAG: --pod-manifest-path="" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338080 2585 flags.go:64] FLAG: --pod-max-pids="-1" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338083 2585 flags.go:64] FLAG: --pods-per-core="0" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338088 2585 flags.go:64] FLAG: --port="10250" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338091 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338094 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00ccd8e79406d8d64" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338097 2585 flags.go:64] FLAG: --qos-reserved="" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338100 2585 flags.go:64] FLAG: --read-only-port="10255" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338103 2585 flags.go:64] FLAG: --register-node="true" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338106 2585 flags.go:64] FLAG: --register-with-taints="" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338110 2585 flags.go:64] FLAG: --registry-burst="10" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338113 2585 flags.go:64] FLAG: --registry-qps="5" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338116 2585 flags.go:64] FLAG: --reserved-cpus="" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338118 2585 flags.go:64] FLAG: --reserved-memory="" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338122 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338125 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338128 2585 flags.go:64] FLAG: --rotate-certificates="false" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338131 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 19 12:25:06.344694 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338134 2585 flags.go:64] FLAG: --runonce="false" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338137 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338141 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338145 2585 flags.go:64] FLAG: --seccomp-default="false" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338147 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338150 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338153 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338156 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338159 2585 flags.go:64] FLAG: --storage-driver-password="root" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338162 2585 flags.go:64] FLAG: --storage-driver-secure="false" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338165 2585 flags.go:64] FLAG: --storage-driver-table="stats" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338168 2585 flags.go:64] FLAG: --storage-driver-user="root" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338171 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338174 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338176 2585 flags.go:64] FLAG: --system-cgroups="" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338179 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338185 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338188 2585 flags.go:64] FLAG: --tls-cert-file="" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338191 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338195 2585 flags.go:64] FLAG: --tls-min-version="" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338198 2585 flags.go:64] FLAG: --tls-private-key-file="" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338200 2585 flags.go:64] FLAG: --topology-manager-policy="none" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338203 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338206 2585 flags.go:64] FLAG: --topology-manager-scope="container" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338209 2585 flags.go:64] FLAG: --v="2" Sep 19 12:25:06.345314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338213 2585 flags.go:64] FLAG: --version="false" Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338217 2585 flags.go:64] FLAG: --vmodule="" Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338221 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.338224 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338359 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338363 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338366 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338368 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338371 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338376 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338379 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338381 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338385 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338389 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338392 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338396 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338399 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338401 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338404 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338407 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 12:25:06.345984 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338410 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338413 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338415 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338418 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338425 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338428 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338431 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338433 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338436 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338439 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338441 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338444 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338447 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338450 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338453 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338455 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338458 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338460 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338463 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338466 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 12:25:06.346532 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338468 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338472 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338475 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338477 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338480 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338482 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338485 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338488 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338490 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338493 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338497 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338499 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338502 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338505 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338507 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338510 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338514 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338518 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338521 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 12:25:06.347066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338524 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338527 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338529 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338532 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338534 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338537 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338539 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338542 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338545 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338548 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338550 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338553 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338555 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338558 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338562 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338565 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338567 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338570 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338572 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338575 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 12:25:06.347528 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338577 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338580 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338582 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338585 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338588 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338590 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338593 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338595 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338598 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338603 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338606 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338609 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338611 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338614 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338617 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338619 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338623 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338627 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338630 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338633 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 12:25:06.348075 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338636 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338639 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338641 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.338644 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.339286 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.346853 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.346870 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.346892 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.346897 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346956 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346961 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346964 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346966 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346969 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 12:25:06.348572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346972 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346977 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346981 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346984 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346987 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346990 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346993 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346996 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.346999 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347002 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347005 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347008 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347011 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347014 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347016 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347020 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347023 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347026 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347029 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 12:25:06.348940 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347031 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347034 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347036 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347039 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347041 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347046 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347049 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347053 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347057 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347060 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347063 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347066 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347070 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347073 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347076 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347079 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347082 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347085 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347088 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 12:25:06.349397 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347091 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347094 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347096 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347099 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347102 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347105 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347108 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347111 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347113 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347116 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347119 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347122 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347125 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347128 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347131 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347134 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347136 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347139 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347142 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347145 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 12:25:06.349902 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347148 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347150 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347152 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347155 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347158 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347160 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347163 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347165 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347168 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347170 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347173 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347176 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347179 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347181 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347184 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347186 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347189 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347191 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347194 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347196 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 12:25:06.350414 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347199 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347202 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347205 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347207 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347210 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347213 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347216 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347218 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347221 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347223 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347226 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347228 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347231 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347234 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347236 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 12:25:06.350951 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347239 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.347244 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347358 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347364 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347367 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347369 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347372 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347375 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347378 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347380 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347383 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347385 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347388 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347391 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347393 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 12:25:06.351319 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347397 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347400 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347404 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347406 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347410 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347413 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347416 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347418 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347422 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347425 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347428 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347431 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347434 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347437 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347439 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347442 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347445 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347447 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347451 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347453 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 12:25:06.351968 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347456 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347458 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347461 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347463 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347466 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347468 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347471 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347473 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347476 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347479 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347482 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347486 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347489 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347491 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347494 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347496 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347499 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347502 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347504 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347507 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 12:25:06.352826 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347510 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347513 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347516 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347518 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347521 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347523 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347526 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347528 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347532 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347534 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347537 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347540 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347542 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347545 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347548 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347550 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347553 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347556 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347559 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 12:25:06.353346 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347561 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347564 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347566 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347569 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347572 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347574 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347577 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347579 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347582 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347585 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347588 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347591 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347595 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347598 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347600 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347603 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347606 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347608 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347611 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 12:25:06.353978 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347613 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347616 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347618 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347621 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347623 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347626 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347628 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:06.347631 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.347636 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.347781 2585 server.go:962] "Client rotation is on, will bootstrap in background" Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.350465 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.351542 2585 server.go:1019] "Starting client certificate rotation" Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.351664 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.351729 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 19 12:25:06.354497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.354136 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 19 12:25:06.355760 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.355740 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 19 12:25:06.360593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.360571 2585 log.go:25] "Validated CRI v1 runtime API" Sep 19 12:25:06.362144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.362125 2585 log.go:25] "Validated CRI v1 image API" Sep 19 12:25:06.363041 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.363020 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 19 12:25:06.366342 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.366320 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b19a7f0c-19c6-4b5b-bd81-ed23676727ce:/dev/nvme0n1p3 bfe80cd7-a746-4620-aef7-1a5efca88539:/dev/nvme0n1p4] Sep 19 12:25:06.366419 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.366340 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 19 12:25:06.371765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.371626 2585 manager.go:216] Machine: {Timestamp:2026-09-19 12:25:06.371080809 +0000 UTC m=+1.977931399 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113290 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c7089958c5165cb7c408066b7375 SystemUUID:ec29c708-9958-c516-5cb7-c408066b7375 BootID:78511b2b-6269-4022-ab91-d8676d127f99 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:1e:ed:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:1e:ed:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:c3:bc:8e:7a:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 19 12:25:06.371765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.371749 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 19 12:25:06.371912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.371834 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 19 12:25:06.373592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.373567 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 19 12:25:06.373768 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.373596 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 19 12:25:06.374796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.374782 2585 topology_manager.go:138] "Creating topology manager with none policy" Sep 19 12:25:06.374796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.374797 2585 container_manager_linux.go:309] "Creating device plugin manager" Sep 19 12:25:06.374904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.374805 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 19 12:25:06.374904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.374817 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 19 12:25:06.374904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.374895 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 19 12:25:06.377879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.377866 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 19 12:25:06.377996 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.377986 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 19 12:25:06.378031 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378027 2585 kubelet.go:486] "Attempting to sync node with API server" Sep 19 12:25:06.378059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378045 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 19 12:25:06.378087 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378060 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 19 12:25:06.378087 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378069 2585 kubelet.go:398] "Adding apiserver pod source" Sep 19 12:25:06.378087 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378081 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 19 12:25:06.378492 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378477 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 19 12:25:06.378611 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.378497 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 19 12:25:06.380913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.380900 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 19 12:25:06.384249 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.384232 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 19 12:25:06.384356 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.384263 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 19 12:25:06.385560 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385545 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 19 12:25:06.385629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385567 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 19 12:25:06.385629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385578 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 19 12:25:06.385629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385588 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 19 12:25:06.385629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385602 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 19 12:25:06.385629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385610 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 19 12:25:06.385629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385629 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385637 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385646 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385655 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385698 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385707 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385744 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 19 12:25:06.385811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.385752 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 19 12:25:06.386709 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.386678 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 19 12:25:06.389649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.389636 2585 server.go:1268] "Started kubelet" Sep 19 12:25:06.389772 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.389745 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 19 12:25:06.389828 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.389799 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 19 12:25:06.389871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.389802 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 19 12:25:06.390799 ip-10-0-131-149 systemd[1]: Started Kubernetes Kubelet. Sep 19 12:25:06.391364 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.391173 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 19 12:25:06.422004 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.421975 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 19 12:25:06.422327 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.422281 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 19 12:25:06.422666 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.422532 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 19 12:25:06.422780 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.422764 2585 server.go:310] "Adding debug handlers to kubelet server" Sep 19 12:25:06.423183 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.423156 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 19 12:25:06.427700 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.426501 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-149.ec2.internal.18d6b87317fdaa2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-149.ec2.internal,UID:ip-10-0-131-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-149.ec2.internal,},FirstTimestamp:2026-09-19 12:25:06.38960081 +0000 UTC m=+1.996451431,LastTimestamp:2026-09-19 12:25:06.38960081 +0000 UTC m=+1.996451431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-149.ec2.internal,}" Sep 19 12:25:06.428814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.428793 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 19 12:25:06.428814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.428811 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 19 12:25:06.429021 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.428981 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:06.429507 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429488 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 19 12:25:06.429507 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429504 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 19 12:25:06.429654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429520 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 19 12:25:06.429723 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429653 2585 reconstruct.go:97] "Volume reconstruction finished" Sep 19 12:25:06.429723 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429663 2585 reconciler.go:29] "Reconciler: start to sync state" Sep 19 12:25:06.429800 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429741 2585 factory.go:55] Registering systemd factory Sep 19 12:25:06.429800 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429756 2585 factory.go:223] Registration of the systemd container factory successfully Sep 19 12:25:06.429988 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429975 2585 factory.go:153] Registering CRI-O factory Sep 19 12:25:06.430036 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.429991 2585 factory.go:223] Registration of the crio container factory successfully Sep 19 12:25:06.430093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.430041 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 19 12:25:06.430093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.430064 2585 factory.go:103] Registering Raw factory Sep 19 12:25:06.430093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.430079 2585 manager.go:1196] Started watching for new ooms in manager Sep 19 12:25:06.430448 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.430432 2585 manager.go:349] Starting recovery of all containers Sep 19 12:25:06.433478 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.433440 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2kr4" Sep 19 12:25:06.438189 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.438052 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 19 12:25:06.438306 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.438271 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 19 12:25:06.439670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.439656 2585 manager.go:354] Recovery completed Sep 19 12:25:06.440665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.440646 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2kr4" Sep 19 12:25:06.446200 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.446187 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.452467 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.452453 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.452530 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.452480 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.452530 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.452490 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.453062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453047 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 19 12:25:06.453062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453063 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 19 12:25:06.453173 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453054 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.453173 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453081 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 19 12:25:06.453955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453940 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.454032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453964 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.454032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.453974 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.454219 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.454204 2585 policy_none.go:49] "None policy: Start" Sep 19 12:25:06.454286 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.454225 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 19 12:25:06.454286 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.454239 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 19 12:25:06.454375 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.454295 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-149.ec2.internal.18d6b8731bbcedcd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-149.ec2.internal,UID:ip-10-0-131-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-149.ec2.internal,},FirstTimestamp:2026-09-19 12:25:06.452467149 +0000 UTC m=+2.059317719,LastTimestamp:2026-09-19 12:25:06.452467149 +0000 UTC m=+2.059317719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-149.ec2.internal,}" Sep 19 12:25:06.455292 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.455281 2585 policy_none.go:47] "Start" Sep 19 12:25:06.504341 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504323 2585 manager.go:337] "Starting Device Plugin manager" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.504370 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504383 2585 server.go:87] "Starting device plugin registration server" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504747 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504769 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504846 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504916 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 19 12:25:06.505550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.504925 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 19 12:25:06.505872 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.505720 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 19 12:25:06.505872 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.505767 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:06.569897 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.569858 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 19 12:25:06.571215 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.571192 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 19 12:25:06.571336 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.571258 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 19 12:25:06.571336 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.571289 2585 kubelet.go:2448] "Starting kubelet main sync loop" Sep 19 12:25:06.571917 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.571871 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 19 12:25:06.573990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.573963 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 12:25:06.605850 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.605761 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.607312 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.607293 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.607437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.607331 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.607437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.607347 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.607437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.607378 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.619262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.619241 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.619341 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.619264 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-149.ec2.internal\": node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:06.672237 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.672175 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal"] Sep 19 12:25:06.672410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.672288 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.674999 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.674910 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.674999 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.674952 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.674999 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.674968 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.676319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.676307 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.676454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.676435 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.676494 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.676488 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.677242 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.677223 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.677351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.677251 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.677351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.677261 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.677351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.677227 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.677351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.677325 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.677351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.677336 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.678468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.678456 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.678523 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.678478 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 12:25:06.679254 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.679241 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientMemory" Sep 19 12:25:06.679332 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.679270 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 12:25:06.679332 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.679283 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeHasSufficientPID" Sep 19 12:25:06.691015 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.690999 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:06.702453 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.702427 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-149.ec2.internal\" not found" node="ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.707011 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.706994 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-149.ec2.internal\" not found" node="ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.731878 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.731843 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f0b85bec9d68f308ab6b1b2c2e08fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal\" (UID: \"02f0b85bec9d68f308ab6b1b2c2e08fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.731878 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.731879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08bdc3a69a1fdcb162b43b9319667a66-config\") pod \"kube-apiserver-proxy-ip-10-0-131-149.ec2.internal\" (UID: \"08bdc3a69a1fdcb162b43b9319667a66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.732072 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.731901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f0b85bec9d68f308ab6b1b2c2e08fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal\" (UID: \"02f0b85bec9d68f308ab6b1b2c2e08fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.791553 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.791518 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:06.832927 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.832897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f0b85bec9d68f308ab6b1b2c2e08fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal\" (UID: \"02f0b85bec9d68f308ab6b1b2c2e08fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.833012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.832936 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f0b85bec9d68f308ab6b1b2c2e08fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal\" (UID: \"02f0b85bec9d68f308ab6b1b2c2e08fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.833012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.832954 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08bdc3a69a1fdcb162b43b9319667a66-config\") pod \"kube-apiserver-proxy-ip-10-0-131-149.ec2.internal\" (UID: \"08bdc3a69a1fdcb162b43b9319667a66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.833012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.832983 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08bdc3a69a1fdcb162b43b9319667a66-config\") pod \"kube-apiserver-proxy-ip-10-0-131-149.ec2.internal\" (UID: \"08bdc3a69a1fdcb162b43b9319667a66\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.833012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.832985 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f0b85bec9d68f308ab6b1b2c2e08fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal\" (UID: \"02f0b85bec9d68f308ab6b1b2c2e08fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.833012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:06.832985 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f0b85bec9d68f308ab6b1b2c2e08fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal\" (UID: \"02f0b85bec9d68f308ab6b1b2c2e08fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:06.892418 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.892334 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:06.992900 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:06.992846 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.006042 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.006016 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:07.010729 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.010420 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" Sep 19 12:25:07.093854 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.093811 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.194396 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.194304 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.294864 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.294832 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.351549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.351516 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 19 12:25:07.352170 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.351719 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 19 12:25:07.395116 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.395067 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.446330 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.446237 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 12:20:06 +0000 UTC" deadline="2028-06-27 18:41:37.737839112 +0000 UTC" Sep 19 12:25:07.446330 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.446276 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15534h16m30.291567245s" Sep 19 12:25:07.449276 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.449246 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 12:25:07.496075 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.496051 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.567346 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.567305 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 12:25:07.596417 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.596394 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.631327 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:07.631296 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f0b85bec9d68f308ab6b1b2c2e08fc.slice/crio-f3b1ba8904d48dc30087f54025faeab7fac0020fffc17033854c3c64102d69b8 WatchSource:0}: Error finding container f3b1ba8904d48dc30087f54025faeab7fac0020fffc17033854c3c64102d69b8: Status 404 returned error can't find the container with id f3b1ba8904d48dc30087f54025faeab7fac0020fffc17033854c3c64102d69b8 Sep 19 12:25:07.631795 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:07.631775 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bdc3a69a1fdcb162b43b9319667a66.slice/crio-686e7935b7880f26b3da19c1a019840d3c9789c7eeadaff5d730b8fe4df8270c WatchSource:0}: Error finding container 686e7935b7880f26b3da19c1a019840d3c9789c7eeadaff5d730b8fe4df8270c: Status 404 returned error can't find the container with id 686e7935b7880f26b3da19c1a019840d3c9789c7eeadaff5d730b8fe4df8270c Sep 19 12:25:07.638243 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.638227 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 12:25:07.697592 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.697493 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.798042 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.798012 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.898913 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:07.898884 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-149.ec2.internal\" not found" Sep 19 12:25:07.984752 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:07.984661 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 12:25:08.029390 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.029366 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" Sep 19 12:25:08.042777 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.042758 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 19 12:25:08.044360 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.044337 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" Sep 19 12:25:08.050173 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.050157 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 19 12:25:08.380115 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.379891 2585 apiserver.go:52] "Watching apiserver" Sep 19 12:25:08.387593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.387569 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 19 12:25:08.387976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.387958 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-jj7nj","openshift-network-diagnostics/network-check-target-rdzch","kube-system/global-pull-secret-syncer-9s4cm","kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn","openshift-multus/network-metrics-daemon-xxhb9","openshift-network-operator/iptables-alerter-f7f4m","openshift-ovn-kubernetes/ovnkube-node-qtjpc","kube-system/konnectivity-agent-vl828","openshift-cluster-node-tuning-operator/tuned-rhsvq","openshift-dns/node-resolver-2nn4d","openshift-image-registry/node-ca-5b7xp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal","openshift-multus/multus-8wc5j"] Sep 19 12:25:08.389904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.389884 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.390654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.390637 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.391909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.391888 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.392002 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.391963 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:08.392609 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.392590 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.392710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.392595 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.393110 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.393094 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.393871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.393850 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.393966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.393915 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ghd89\"" Sep 19 12:25:08.395268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.394103 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.395268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.394331 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m8pst\"" Sep 19 12:25:08.405901 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.405881 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.406279 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.406259 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.407021 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.406999 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:08.407121 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.407074 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:08.407259 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.407243 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.409122 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.407861 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.409122 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.407908 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.410238 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410210 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410408 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410446 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2g2rt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410476 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410639 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410659 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8wf5x\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410217 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410837 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9k2xd\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.410862 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.410939 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411045 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411106 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jmmh4\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411199 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411267 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411289 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411518 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411646 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411825 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.411887 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 19 12:25:08.412165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.412012 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rp2g4\"" Sep 19 12:25:08.413062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.412821 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.413124 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.413619 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.414275 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.414345 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.414462 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.414493 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.414558 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m8g88\"" Sep 19 12:25:08.414799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.414659 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 19 12:25:08.416549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.415978 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 19 12:25:08.416549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.416105 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 19 12:25:08.416549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.416356 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 19 12:25:08.416910 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.416891 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d9vjp\"" Sep 19 12:25:08.431312 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.431295 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 19 12:25:08.431599 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.431584 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 19 12:25:08.441184 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.440994 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442527 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20e62891-9f88-4a58-abc5-2694ae518b2d-tmp\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-etc-kubernetes\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442588 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-ovn\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442621 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-host\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442634 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-kubelet\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442654 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442719 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442760 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-kubernetes\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-systemd\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-k8s-cni-cncf-io\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442823 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-cni-bin\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-node-log\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442849 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovn-node-metrics-cert\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444099 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442864 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23e2cb5c-9d8d-484d-8553-921c7016a744-iptables-alerter-script\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a901150b-7247-4fb6-8577-e0bc8d519a2a-cni-binary-copy\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442906 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442935 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5502ec92-c349-4d1d-b324-c14f8e73841f-agent-certs\") pod \"konnectivity-agent-vl828\" (UID: \"5502ec92-c349-4d1d-b324-c14f8e73841f\") " pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442976 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-cni-netd\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.442991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-cnibin\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443018 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-sys\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443031 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-var-lib-kubelet\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443045 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-run-netns\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-log-socket\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-dbus\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443102 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-serviceca\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-tmp-dir\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tckgc\" (UniqueName: \"kubernetes.io/projected/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-kube-api-access-tckgc\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.444876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysctl-d\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-hosts-file\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443187 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-system-cni-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-conf-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5502ec92-c349-4d1d-b324-c14f8e73841f-konnectivity-ca\") pod \"konnectivity-agent-vl828\" (UID: \"5502ec92-c349-4d1d-b324-c14f8e73841f\") " pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443235 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovnkube-config\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443251 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovnkube-script-lib\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-cnibin\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443299 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-modprobe-d\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-hostroot\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443332 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-kubelet\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443392 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjtf\" (UniqueName: \"kubernetes.io/projected/9691988f-8bb1-4962-9649-8baac63318bf-kube-api-access-shjtf\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443406 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rnf\" (UniqueName: \"kubernetes.io/projected/23e2cb5c-9d8d-484d-8553-921c7016a744-kube-api-access-h5rnf\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443421 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbk7\" (UniqueName: \"kubernetes.io/projected/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-kube-api-access-ffbk7\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.445632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443437 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-lib-modules\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-daemon-config\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8g4h\" (UniqueName: \"kubernetes.io/projected/a901150b-7247-4fb6-8577-e0bc8d519a2a-kube-api-access-h8g4h\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443480 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-etc-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-cni-bin\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443508 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-cni-multus\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443522 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgsj\" (UniqueName: \"kubernetes.io/projected/5f2e3995-83ae-49ff-b232-c562960cc94d-kube-api-access-mfgsj\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443553 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443567 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23e2cb5c-9d8d-484d-8553-921c7016a744-host-slash\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-host\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443593 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysconfig\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443606 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysctl-conf\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443623 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-netns\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443638 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-multus-certs\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-device-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.446144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443668 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7659\" (UniqueName: \"kubernetes.io/projected/177114ca-4aae-4bdc-bfca-35df7eaed14f-kube-api-access-j7659\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443702 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-kubelet-config\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443727 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443754 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjmk\" (UniqueName: \"kubernetes.io/projected/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-kube-api-access-8pjmk\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-run\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7d99\" (UniqueName: \"kubernetes.io/projected/20e62891-9f88-4a58-abc5-2694ae518b2d-kube-api-access-r7d99\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.443856 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-cni-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444010 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-slash\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-systemd\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-tuned\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444152 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-socket-dir-parent\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-systemd-units\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444305 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-var-lib-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444328 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-env-overrides\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-system-cni-dir\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.446713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.444389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.463483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.463458 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7972" Sep 19 12:25:08.468631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.468612 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7972" Sep 19 12:25:08.545482 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-etc-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-cni-bin\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-cni-multus\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545532 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgsj\" (UniqueName: \"kubernetes.io/projected/5f2e3995-83ae-49ff-b232-c562960cc94d-kube-api-access-mfgsj\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23e2cb5c-9d8d-484d-8553-921c7016a744-host-slash\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.545627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-host\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysconfig\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545657 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysctl-conf\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545680 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-netns\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545721 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-multus-certs\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545743 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-device-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7659\" (UniqueName: \"kubernetes.io/projected/177114ca-4aae-4bdc-bfca-35df7eaed14f-kube-api-access-j7659\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-kubelet-config\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545816 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545839 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjmk\" (UniqueName: \"kubernetes.io/projected/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-kube-api-access-8pjmk\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545864 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-run\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7d99\" (UniqueName: \"kubernetes.io/projected/20e62891-9f88-4a58-abc5-2694ae518b2d-kube-api-access-r7d99\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545910 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-cni-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545934 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-slash\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-systemd\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.545979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-tuned\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546004 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-socket-dir-parent\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.546055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-systemd-units\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-var-lib-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546074 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-env-overrides\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546130 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-system-cni-dir\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546182 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20e62891-9f88-4a58-abc5-2694ae518b2d-tmp\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-etc-kubernetes\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546274 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-ovn\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546296 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-host\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-kubelet\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-kubernetes\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-systemd\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.546879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-k8s-cni-cncf-io\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546463 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-cni-bin\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-node-log\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546515 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovn-node-metrics-cert\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23e2cb5c-9d8d-484d-8553-921c7016a744-iptables-alerter-script\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546571 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a901150b-7247-4fb6-8577-e0bc8d519a2a-cni-binary-copy\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546647 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5502ec92-c349-4d1d-b324-c14f8e73841f-agent-certs\") pod \"konnectivity-agent-vl828\" (UID: \"5502ec92-c349-4d1d-b324-c14f8e73841f\") " pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-cni-netd\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-cnibin\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546828 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-sys\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-var-lib-kubelet\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546883 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-run-netns\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546908 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-log-socket\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.547721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-dbus\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-serviceca\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.546983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-tmp-dir\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tckgc\" (UniqueName: \"kubernetes.io/projected/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-kube-api-access-tckgc\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysctl-d\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547087 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-hosts-file\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-system-cni-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-conf-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547233 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5502ec92-c349-4d1d-b324-c14f8e73841f-konnectivity-ca\") pod \"konnectivity-agent-vl828\" (UID: \"5502ec92-c349-4d1d-b324-c14f8e73841f\") " pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547259 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovnkube-config\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovnkube-script-lib\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-cnibin\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-modprobe-d\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-hostroot\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-kubelet\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.548491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547488 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjtf\" (UniqueName: \"kubernetes.io/projected/9691988f-8bb1-4962-9649-8baac63318bf-kube-api-access-shjtf\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547515 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rnf\" (UniqueName: \"kubernetes.io/projected/23e2cb5c-9d8d-484d-8553-921c7016a744-kube-api-access-h5rnf\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbk7\" (UniqueName: \"kubernetes.io/projected/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-kube-api-access-ffbk7\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547569 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-lib-modules\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-daemon-config\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.547625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8g4h\" (UniqueName: \"kubernetes.io/projected/a901150b-7247-4fb6-8577-e0bc8d519a2a-kube-api-access-h8g4h\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548029 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-etc-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-cni-bin\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-cni-multus\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548227 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-socket-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548381 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-cni-bin\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548482 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-node-log\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.548866 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549043 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23e2cb5c-9d8d-484d-8553-921c7016a744-host-slash\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549163 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-host\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.549211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549213 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysconfig\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.549778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549339 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysctl-conf\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.549778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549379 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-netns\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.549778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549419 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-multus-certs\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.549778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549476 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-device-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.549778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549742 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-kubelet-config\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.550007 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-hostroot\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.550007 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.549818 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:08.550007 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549841 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-sysctl-d\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.550007 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.549885 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:25:09.049866488 +0000 UTC m=+4.656717047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:08.550007 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.549978 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-kubelet\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550244 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550056 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-run\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.550244 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550169 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.550244 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550220 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550317 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5502ec92-c349-4d1d-b324-c14f8e73841f-konnectivity-ca\") pod \"konnectivity-agent-vl828\" (UID: \"5502ec92-c349-4d1d-b324-c14f8e73841f\") " pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.550396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550366 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovnkube-config\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550391 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-hosts-file\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.550541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550435 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-systemd\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550415 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23e2cb5c-9d8d-484d-8553-921c7016a744-iptables-alerter-script\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.550541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550448 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-system-cni-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.550541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550436 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-slash\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550487 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-cnibin\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.550541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550547 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-var-lib-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550556 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-conf-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-etc-kubernetes\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550623 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-registration-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-cni-dir\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-sys-fs\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-systemd-units\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.550856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.551229 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551102 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovnkube-script-lib\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.551229 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551115 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a901150b-7247-4fb6-8577-e0bc8d519a2a-cni-binary-copy\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.551229 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551195 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-modprobe-d\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.551373 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.550505 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-socket-dir-parent\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551587 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-tmp-dir\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551661 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9691988f-8bb1-4962-9649-8baac63318bf-system-cni-dir\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551714 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-openvswitch\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-run-ovn\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-host\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-var-lib-kubelet\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551847 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-run-netns\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551847 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-lib-modules\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.551883 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f2e3995-83ae-49ff-b232-c562960cc94d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/177114ca-4aae-4bdc-bfca-35df7eaed14f-env-overrides\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552242 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552277 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a901150b-7247-4fb6-8577-e0bc8d519a2a-multus-daemon-config\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552290 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-sys\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552311 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-host-cni-netd\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552323 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-host-run-k8s-cni-cncf-io\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/177114ca-4aae-4bdc-bfca-35df7eaed14f-log-socket\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552330 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9691988f-8bb1-4962-9649-8baac63318bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.552886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552367 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-var-lib-kubelet\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-kubernetes\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-systemd\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552414 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-dbus\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552425 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a901150b-7247-4fb6-8577-e0bc8d519a2a-cnibin\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.552500 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.552574 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:25:09.052558761 +0000 UTC m=+4.659409333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552765 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-serviceca\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.552961 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20e62891-9f88-4a58-abc5-2694ae518b2d-etc-tuned\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.553913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.553589 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5502ec92-c349-4d1d-b324-c14f8e73841f-agent-certs\") pod \"konnectivity-agent-vl828\" (UID: \"5502ec92-c349-4d1d-b324-c14f8e73841f\") " pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.554584 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.554541 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/177114ca-4aae-4bdc-bfca-35df7eaed14f-ovn-node-metrics-cert\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.554805 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.554781 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20e62891-9f88-4a58-abc5-2694ae518b2d-tmp\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.555791 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.555773 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8g4h\" (UniqueName: \"kubernetes.io/projected/a901150b-7247-4fb6-8577-e0bc8d519a2a-kube-api-access-h8g4h\") pod \"multus-8wc5j\" (UID: \"a901150b-7247-4fb6-8577-e0bc8d519a2a\") " pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.557739 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.557702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7659\" (UniqueName: \"kubernetes.io/projected/177114ca-4aae-4bdc-bfca-35df7eaed14f-kube-api-access-j7659\") pod \"ovnkube-node-qtjpc\" (UID: \"177114ca-4aae-4bdc-bfca-35df7eaed14f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.558223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.558196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgsj\" (UniqueName: \"kubernetes.io/projected/5f2e3995-83ae-49ff-b232-c562960cc94d-kube-api-access-mfgsj\") pod \"aws-ebs-csi-driver-node-4zjcn\" (UID: \"5f2e3995-83ae-49ff-b232-c562960cc94d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.558336 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.558316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjmk\" (UniqueName: \"kubernetes.io/projected/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-kube-api-access-8pjmk\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:08.559028 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.558966 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjtf\" (UniqueName: \"kubernetes.io/projected/9691988f-8bb1-4962-9649-8baac63318bf-kube-api-access-shjtf\") pod \"multus-additional-cni-plugins-jj7nj\" (UID: \"9691988f-8bb1-4962-9649-8baac63318bf\") " pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.561414 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.561343 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:08.561414 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.561371 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:08.561414 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.561385 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:08.561629 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:08.561453 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:09.061432635 +0000 UTC m=+4.668283209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:08.562714 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.562136 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7d99\" (UniqueName: \"kubernetes.io/projected/20e62891-9f88-4a58-abc5-2694ae518b2d-kube-api-access-r7d99\") pod \"tuned-rhsvq\" (UID: \"20e62891-9f88-4a58-abc5-2694ae518b2d\") " pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.563722 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.563661 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tckgc\" (UniqueName: \"kubernetes.io/projected/d62b4d7a-0062-4806-ba22-6fc3dc29e3d2-kube-api-access-tckgc\") pod \"node-resolver-2nn4d\" (UID: \"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2\") " pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.564097 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.564070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rnf\" (UniqueName: \"kubernetes.io/projected/23e2cb5c-9d8d-484d-8553-921c7016a744-kube-api-access-h5rnf\") pod \"iptables-alerter-f7f4m\" (UID: \"23e2cb5c-9d8d-484d-8553-921c7016a744\") " pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.564183 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.564130 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbk7\" (UniqueName: \"kubernetes.io/projected/7d3187f2-8b12-441d-bc4f-17bcaa9b12e9-kube-api-access-ffbk7\") pod \"node-ca-5b7xp\" (UID: \"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9\") " pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:08.579410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.579360 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" event={"ID":"08bdc3a69a1fdcb162b43b9319667a66","Type":"ContainerStarted","Data":"686e7935b7880f26b3da19c1a019840d3c9789c7eeadaff5d730b8fe4df8270c"} Sep 19 12:25:08.580883 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.580838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" event={"ID":"02f0b85bec9d68f308ab6b1b2c2e08fc","Type":"ContainerStarted","Data":"f3b1ba8904d48dc30087f54025faeab7fac0020fffc17033854c3c64102d69b8"} Sep 19 12:25:08.719653 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.719593 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" Sep 19 12:25:08.727532 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.727511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2nn4d" Sep 19 12:25:08.736212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.736194 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8wc5j" Sep 19 12:25:08.738386 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.738365 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 12:25:08.745227 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.745206 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" Sep 19 12:25:08.756005 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.755988 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f7f4m" Sep 19 12:25:08.761649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.761631 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:08.768348 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.768329 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:08.782890 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.782868 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" Sep 19 12:25:08.789546 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:08.789527 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5b7xp" Sep 19 12:25:09.050666 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.050586 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:09.050815 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.050741 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:09.050815 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.050806 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:25:10.050785965 +0000 UTC m=+5.657636530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:09.151644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.151599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:09.151816 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.151666 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:09.151816 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.151734 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:09.151816 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.151784 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:09.151816 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.151800 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:09.151816 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.151810 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:25:10.151789826 +0000 UTC m=+5.758640385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:09.151816 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.151812 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:09.152070 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.151866 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:10.151851511 +0000 UTC m=+5.758702070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:09.470818 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.470705 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 12:20:08 +0000 UTC" deadline="2028-03-30 22:10:16.922937369 +0000 UTC" Sep 19 12:25:09.470818 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.470769 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13401h45m7.452177776s" Sep 19 12:25:09.549080 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:09.549057 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2e3995_83ae_49ff_b232_c562960cc94d.slice/crio-184acaa7f289f42d063ccb2812f1823f316b474a639ede6b9336507881cdb640 WatchSource:0}: Error finding container 184acaa7f289f42d063ccb2812f1823f316b474a639ede6b9336507881cdb640: Status 404 returned error can't find the container with id 184acaa7f289f42d063ccb2812f1823f316b474a639ede6b9336507881cdb640 Sep 19 12:25:09.551400 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:09.551368 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62b4d7a_0062_4806_ba22_6fc3dc29e3d2.slice/crio-584d42d38de7fbcf2d46f2398ee636b0a926e56f3fff485d07a47727daa966c9 WatchSource:0}: Error finding container 584d42d38de7fbcf2d46f2398ee636b0a926e56f3fff485d07a47727daa966c9: Status 404 returned error can't find the container with id 584d42d38de7fbcf2d46f2398ee636b0a926e56f3fff485d07a47727daa966c9 Sep 19 12:25:09.552060 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:09.552009 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e62891_9f88_4a58_abc5_2694ae518b2d.slice/crio-8aeaa96e52a7ced6d93ffa823bdecab27738c9a787fe26431f76298b1cbfc511 WatchSource:0}: Error finding container 8aeaa96e52a7ced6d93ffa823bdecab27738c9a787fe26431f76298b1cbfc511: Status 404 returned error can't find the container with id 8aeaa96e52a7ced6d93ffa823bdecab27738c9a787fe26431f76298b1cbfc511 Sep 19 12:25:09.555469 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:09.555447 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda901150b_7247_4fb6_8577_e0bc8d519a2a.slice/crio-1a30bffeb28f9d7017f8cfc27866c0408eda7a478e0324b74eab927540f34295 WatchSource:0}: Error finding container 1a30bffeb28f9d7017f8cfc27866c0408eda7a478e0324b74eab927540f34295: Status 404 returned error can't find the container with id 1a30bffeb28f9d7017f8cfc27866c0408eda7a478e0324b74eab927540f34295 Sep 19 12:25:09.556617 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:09.556598 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177114ca_4aae_4bdc_bfca_35df7eaed14f.slice/crio-8765ae2ed2a368c755752b230fa310c1fc570d20449b7835ae391cabbfeb8db3 WatchSource:0}: Error finding container 8765ae2ed2a368c755752b230fa310c1fc570d20449b7835ae391cabbfeb8db3: Status 404 returned error can't find the container with id 8765ae2ed2a368c755752b230fa310c1fc570d20449b7835ae391cabbfeb8db3 Sep 19 12:25:09.571918 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.571896 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:09.572067 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:09.572019 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:09.583621 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.583438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"8765ae2ed2a368c755752b230fa310c1fc570d20449b7835ae391cabbfeb8db3"} Sep 19 12:25:09.584495 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.584477 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8wc5j" event={"ID":"a901150b-7247-4fb6-8577-e0bc8d519a2a","Type":"ContainerStarted","Data":"1a30bffeb28f9d7017f8cfc27866c0408eda7a478e0324b74eab927540f34295"} Sep 19 12:25:09.585613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.585595 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2nn4d" event={"ID":"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2","Type":"ContainerStarted","Data":"584d42d38de7fbcf2d46f2398ee636b0a926e56f3fff485d07a47727daa966c9"} Sep 19 12:25:09.586805 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.586779 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5b7xp" event={"ID":"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9","Type":"ContainerStarted","Data":"319a072e8b44a5fb6055b3095ba8e251536b14a3cbe8ac0a94c31e23e2b02e57"} Sep 19 12:25:09.587966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.587947 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerStarted","Data":"48f29d0359c3f7042c9794b81c6126e6c9071a3180ba21995bf14e1c53dc0ac6"} Sep 19 12:25:09.589354 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.589335 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f7f4m" event={"ID":"23e2cb5c-9d8d-484d-8553-921c7016a744","Type":"ContainerStarted","Data":"b6a4cab423e9b4eea31fd9b962db38d1fc20d1c391b14f8cea8c9ce37ac24c11"} Sep 19 12:25:09.590598 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.590576 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" event={"ID":"20e62891-9f88-4a58-abc5-2694ae518b2d","Type":"ContainerStarted","Data":"8aeaa96e52a7ced6d93ffa823bdecab27738c9a787fe26431f76298b1cbfc511"} Sep 19 12:25:09.595168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.595147 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" event={"ID":"5f2e3995-83ae-49ff-b232-c562960cc94d","Type":"ContainerStarted","Data":"184acaa7f289f42d063ccb2812f1823f316b474a639ede6b9336507881cdb640"} Sep 19 12:25:09.599697 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:09.599666 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vl828" event={"ID":"5502ec92-c349-4d1d-b324-c14f8e73841f","Type":"ContainerStarted","Data":"b46a883c71f464b7e568c5894d53328e5e6a624138cfa64bcf880cbcdeab65ca"} Sep 19 12:25:10.057063 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.056892 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:10.057063 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.057053 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:10.057273 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.057141 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:25:12.05710593 +0000 UTC m=+7.663956501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:10.157807 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.157771 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:10.157959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.157829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:10.158029 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.157977 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:10.158029 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.157995 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:10.158029 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.158007 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:10.158175 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.158061 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:12.158043834 +0000 UTC m=+7.764894393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:10.158454 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.158436 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:10.158534 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.158492 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:25:12.158476284 +0000 UTC m=+7.765326853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:10.471175 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.471061 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 12:20:08 +0000 UTC" deadline="2028-03-02 23:51:03.767552566 +0000 UTC" Sep 19 12:25:10.471175 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.471117 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12731h25m53.296439052s" Sep 19 12:25:10.574477 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.571624 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:10.574477 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.571769 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:10.574477 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.572143 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:10.574477 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:10.572232 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:10.606999 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.606379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" event={"ID":"08bdc3a69a1fdcb162b43b9319667a66","Type":"ContainerStarted","Data":"6607208b1e6c6e920c3ab99fa70827ccb72607cc017d9df21fa1a19fc4af2c3f"} Sep 19 12:25:10.616638 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.616138 2585 generic.go:356] "Generic (PLEG): container finished" podID="02f0b85bec9d68f308ab6b1b2c2e08fc" containerID="bcd2d5b1622ed361ca58519fa624b7783ee4f2923d457a047b2960ead30b24d3" exitCode=0 Sep 19 12:25:10.616638 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.616180 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" event={"ID":"02f0b85bec9d68f308ab6b1b2c2e08fc","Type":"ContainerDied","Data":"bcd2d5b1622ed361ca58519fa624b7783ee4f2923d457a047b2960ead30b24d3"} Sep 19 12:25:10.633591 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:10.633545 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-149.ec2.internal" podStartSLOduration=2.633527391 podStartE2EDuration="2.633527391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:10.620433603 +0000 UTC m=+6.227284200" watchObservedRunningTime="2026-09-19 12:25:10.633527391 +0000 UTC m=+6.240377974" Sep 19 12:25:11.572334 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:11.572304 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:11.572795 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:11.572429 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:11.635459 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:11.635359 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" event={"ID":"02f0b85bec9d68f308ab6b1b2c2e08fc","Type":"ContainerStarted","Data":"a181adb48a783db4538a13f2f3552b3300471175cd5e7cae8c78617d54b3480c"} Sep 19 12:25:12.074787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:12.074464 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:12.074787 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.074633 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:12.074787 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.074707 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:25:16.074671512 +0000 UTC m=+11.681522078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:12.175057 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:12.175022 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:12.175209 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:12.175083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:12.175288 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.175261 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:12.175288 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.175283 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:12.175387 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.175296 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:12.175387 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.175359 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:16.175333723 +0000 UTC m=+11.782184285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:12.175771 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.175753 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:12.175871 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.175806 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:25:16.175791844 +0000 UTC m=+11.782642408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:12.572796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:12.571836 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:12.572796 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.571969 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:12.572796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:12.572778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:12.573310 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:12.572889 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:13.572072 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:13.572039 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:13.572257 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:13.572158 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:14.575070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:14.574388 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:14.575070 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:14.574527 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:14.575070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:14.574915 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:14.575070 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:14.575006 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:15.572173 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:15.572138 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:15.572342 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:15.572273 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:16.110485 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:16.110441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:16.111054 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.110644 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:16.111054 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.110753 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:25:24.110733452 +0000 UTC m=+19.717584012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:16.211068 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:16.211030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:16.211234 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:16.211096 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:16.211280 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.211250 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:16.211280 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.211267 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:16.211280 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.211280 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:16.211433 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.211338 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:24.21131949 +0000 UTC m=+19.818170051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:16.211721 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.211602 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:16.211721 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.211676 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:25:24.211657412 +0000 UTC m=+19.818507984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:16.573272 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:16.573194 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:16.573415 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.573324 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:16.573843 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:16.573822 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:16.573947 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:16.573930 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:17.572457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:17.572421 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:17.573066 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:17.572556 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:18.572703 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:18.572185 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:18.572703 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:18.572201 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:18.572703 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:18.572315 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:18.572703 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:18.572400 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:19.572293 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:19.572256 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:19.572457 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:19.572386 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:20.571538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:20.571503 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:20.571538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:20.571539 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:20.572084 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:20.571650 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:20.572084 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:20.571805 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:21.572002 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:21.571971 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:21.572407 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:21.572100 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:22.572157 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:22.572117 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:22.572157 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:22.572165 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:22.572585 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:22.572245 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:22.572585 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:22.572369 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:23.572161 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:23.572129 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:23.572351 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:23.572266 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:24.174422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:24.174378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:24.174610 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.174513 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:24.174610 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.174587 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:25:40.174570918 +0000 UTC m=+35.781421481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:24.275670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:24.275638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:24.275855 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:24.275703 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:24.275855 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.275822 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:24.275971 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.275856 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:24.275971 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.275872 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:24.275971 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.275884 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:24.275971 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.275897 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:25:40.275877482 +0000 UTC m=+35.882728043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:24.275971 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.275931 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:40.275917304 +0000 UTC m=+35.882767862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:24.571618 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:24.571532 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:24.571618 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:24.571567 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:24.571856 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.571706 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:24.571856 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:24.571799 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:25.572119 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:25.572082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:25.572570 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:25.572208 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:26.572266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:26.572235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:26.572771 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:26.572351 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:26.572771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:26.572382 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:26.572771 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:26.572511 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:27.571784 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:27.571757 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:27.571909 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:27.571888 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:27.673726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:27.673527 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vl828" event={"ID":"5502ec92-c349-4d1d-b324-c14f8e73841f","Type":"ContainerStarted","Data":"ae827d945e55f4699f29e09aef7c7f978490a342cf453bdd6a784c0725eb7e34"} Sep 19 12:25:27.675755 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:27.675658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2nn4d" event={"ID":"d62b4d7a-0062-4806-ba22-6fc3dc29e3d2","Type":"ContainerStarted","Data":"168dc93bd0b201fc5b6ec2527fa3f11690afdb2d67a74b39a410f45a7dd95992"} Sep 19 12:25:27.690966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:27.690911 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-149.ec2.internal" podStartSLOduration=19.690889468 podStartE2EDuration="19.690889468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:11.650518076 +0000 UTC m=+7.257368676" watchObservedRunningTime="2026-09-19 12:25:27.690889468 +0000 UTC m=+23.297740069" Sep 19 12:25:27.704475 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:27.704422 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vl828" podStartSLOduration=7.742762677 podStartE2EDuration="21.704400851s" totalImagesPullingTime="13.961638174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:27.691848561 +0000 UTC m=+23.298699157" watchObservedRunningTime="2026-09-19 12:25:27.704400851 +0000 UTC m=+23.311251428" Sep 19 12:25:28.572566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.572334 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:28.572773 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.572382 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:28.572773 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:28.572677 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:28.572894 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:28.572774 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:28.679032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.678994 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5b7xp" event={"ID":"7d3187f2-8b12-441d-bc4f-17bcaa9b12e9","Type":"ContainerStarted","Data":"703b8e7c74555b62ce438800827ed49030ed03562f6a145ad18a6d9660d5a33a"} Sep 19 12:25:28.680439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.680418 2585 generic.go:356] "Generic (PLEG): container finished" podID="9691988f-8bb1-4962-9649-8baac63318bf" containerID="5d983544761dd518c2cf7c11cd569a2d4b7838b1dc75fd605d6cf7c016708f37" exitCode=0 Sep 19 12:25:28.680542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.680498 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerDied","Data":"5d983544761dd518c2cf7c11cd569a2d4b7838b1dc75fd605d6cf7c016708f37"} Sep 19 12:25:28.682144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.682116 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" event={"ID":"20e62891-9f88-4a58-abc5-2694ae518b2d","Type":"ContainerStarted","Data":"a0d1244cd7199f4862b8b079c54feba46d3e0d91bcd7054d4cf34bfceaaea2dd"} Sep 19 12:25:28.683541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.683519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" event={"ID":"5f2e3995-83ae-49ff-b232-c562960cc94d","Type":"ContainerStarted","Data":"329a2d03c1868dcd16e5d4c3132e5485c690cc04665256a0aeda798f9f6b747c"} Sep 19 12:25:28.686476 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.686459 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"09ce91e68a08736b579f147ab440d801973c31e0c05ee3c4680ccd2d8c34f34b"} Sep 19 12:25:28.686565 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.686516 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"e4b6c1bc55c7e8dd1f95c5b085773feface0e01fabde3f13f6af761511652bb2"} Sep 19 12:25:28.686565 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.686548 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"dbfa611f8246c915c90587096cab134dbbba2c009dc9d7648f7698a3be2856bb"} Sep 19 12:25:28.686665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.686589 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"69804b46b2ee7bfd836cc4bbcfd9da5de67bd3f28d518474e825348771fac501"} Sep 19 12:25:28.686665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.686639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"1903f5bd2da12fd2225f15698a4124262cfb954a4a9f3b6fa8e677493d3fc3e3"} Sep 19 12:25:28.686762 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.686701 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"5a978837985b89bba4e72ce8fe96aa8173ddd216ce2267f1decca9ad179c662f"} Sep 19 12:25:28.687725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.687700 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8wc5j" event={"ID":"a901150b-7247-4fb6-8577-e0bc8d519a2a","Type":"ContainerStarted","Data":"895fc4f997c00a0a90f524db101b02ba22cbbd713d5d8f5d2fc4f70cf0dc303d"} Sep 19 12:25:28.692584 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.692546 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2nn4d" podStartSLOduration=4.7541178120000005 podStartE2EDuration="22.692533857s" totalImagesPullingTime="17.938416045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:27.704735301 +0000 UTC m=+23.311585901" watchObservedRunningTime="2026-09-19 12:25:28.692533857 +0000 UTC m=+24.299384481" Sep 19 12:25:28.692665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.692620 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5b7xp" podStartSLOduration=3.750974095 podStartE2EDuration="21.692615643s" totalImagesPullingTime="17.941641548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:28.692228342 +0000 UTC m=+24.299078940" watchObservedRunningTime="2026-09-19 12:25:28.692615643 +0000 UTC m=+24.299466238" Sep 19 12:25:28.743321 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.743277 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rhsvq" podStartSLOduration=4.787057038 podStartE2EDuration="22.743263924s" totalImagesPullingTime="17.956206886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:28.726776659 +0000 UTC m=+24.333627269" watchObservedRunningTime="2026-09-19 12:25:28.743263924 +0000 UTC m=+24.350114502" Sep 19 12:25:28.743510 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.743487 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8wc5j" podStartSLOduration=4.754622174 podStartE2EDuration="22.74348109s" totalImagesPullingTime="17.988858916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:28.742927754 +0000 UTC m=+24.349778353" watchObservedRunningTime="2026-09-19 12:25:28.74348109 +0000 UTC m=+24.350331669" Sep 19 12:25:28.808895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:28.808873 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 19 12:25:29.438012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.437984 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:29.438655 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.438630 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:29.519158 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.519058 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T12:25:28.808892398Z","UUID":"9cc9ddba-09b8-4076-bacb-0c4f0719c3ea","Handler":null,"Name":"","Endpoint":""} Sep 19 12:25:29.520840 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.520816 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 19 12:25:29.520975 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.520853 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 19 12:25:29.572432 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.572404 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:29.572548 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:29.572530 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:29.692439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.692365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f7f4m" event={"ID":"23e2cb5c-9d8d-484d-8553-921c7016a744","Type":"ContainerStarted","Data":"8d8b1c38dd1b3d5ed7bd723fd29391e80c5da6d087ce4123f692825702895945"} Sep 19 12:25:29.695052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.695018 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" event={"ID":"5f2e3995-83ae-49ff-b232-c562960cc94d","Type":"ContainerStarted","Data":"d50e3ae2bc11fc25e0e76bd04eda164aa13354df7281254157201ab1fbbbe484"} Sep 19 12:25:29.706297 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:29.706245 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f7f4m" podStartSLOduration=5.777915535 podStartE2EDuration="23.706226801s" totalImagesPullingTime="17.928311266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:29.706143429 +0000 UTC m=+25.312994033" watchObservedRunningTime="2026-09-19 12:25:29.706226801 +0000 UTC m=+25.313077382" Sep 19 12:25:30.571870 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:30.571843 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:30.572044 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:30.571842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:30.572044 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:30.571961 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:30.572164 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:30.572046 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:30.700513 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:30.700462 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"73a0962795b7bdbe378aa07070938553d8537523ab9ef31ad2ed80b9ed2cde4a"} Sep 19 12:25:30.703215 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:30.703188 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" event={"ID":"5f2e3995-83ae-49ff-b232-c562960cc94d","Type":"ContainerStarted","Data":"6a435eeaa850736925d6603a5b2111bb2a3863a850cfd9fe88561d72fe10a8a4"} Sep 19 12:25:30.703391 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:30.703374 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 19 12:25:30.720417 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:30.720364 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zjcn" podStartSLOduration=4.830263157 podStartE2EDuration="24.720345117s" totalImagesPullingTime="19.89008196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:30.7192849 +0000 UTC m=+26.326135499" watchObservedRunningTime="2026-09-19 12:25:30.720345117 +0000 UTC m=+26.327195697" Sep 19 12:25:31.571858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:31.571829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:31.572033 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:31.571946 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:32.572371 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:32.572343 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:32.572371 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:32.572356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:32.572892 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:32.572456 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:32.572892 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:32.572558 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:33.572300 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.572136 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:33.572434 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:33.572375 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:33.712091 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.712025 2585 generic.go:356] "Generic (PLEG): container finished" podID="9691988f-8bb1-4962-9649-8baac63318bf" containerID="236b552f65b00b1add40cb3a38ebcde30483572015af876c4d4070a465c0b7b0" exitCode=0 Sep 19 12:25:33.712192 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.712108 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerDied","Data":"236b552f65b00b1add40cb3a38ebcde30483572015af876c4d4070a465c0b7b0"} Sep 19 12:25:33.716020 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.716000 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" event={"ID":"177114ca-4aae-4bdc-bfca-35df7eaed14f","Type":"ContainerStarted","Data":"00fed3b975478bdda8d2968de21c83747fcb63f3942a1ca6c213e3df08e46ebe"} Sep 19 12:25:33.716260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.716233 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:33.730991 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.730973 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:33.752466 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:33.752434 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" podStartSLOduration=9.747096135 podStartE2EDuration="27.752421955s" totalImagesPullingTime="18.00532582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:33.751321979 +0000 UTC m=+29.358172591" watchObservedRunningTime="2026-09-19 12:25:33.752421955 +0000 UTC m=+29.359272534" Sep 19 12:25:34.572533 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:34.572503 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:34.572533 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:34.572520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:34.573091 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:34.572635 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:34.573091 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:34.572788 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:34.718401 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:34.718370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:34.718562 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:34.718489 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:34.733612 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:34.733587 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:25:35.099529 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.099363 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xxhb9"] Sep 19 12:25:35.099678 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.099661 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:35.099828 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:35.099804 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:35.102185 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.102160 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9s4cm"] Sep 19 12:25:35.102278 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.102265 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:35.102388 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:35.102366 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:35.102765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.102742 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rdzch"] Sep 19 12:25:35.102867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.102840 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:35.102952 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:35.102931 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:35.721558 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.721527 2585 generic.go:356] "Generic (PLEG): container finished" podID="9691988f-8bb1-4962-9649-8baac63318bf" containerID="8a95c27eeac8bcf83b9e0827dfe7dc0b386d50bc458186ec3fde299d6325c852" exitCode=0 Sep 19 12:25:35.721955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:35.721565 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerDied","Data":"8a95c27eeac8bcf83b9e0827dfe7dc0b386d50bc458186ec3fde299d6325c852"} Sep 19 12:25:36.577414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:36.577388 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:36.577646 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:36.577487 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:36.577646 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:36.577512 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:36.577646 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:36.577513 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:36.577646 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:36.577574 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:36.577646 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:36.577639 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:37.158330 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:37.158267 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:37.158798 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:37.158372 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 19 12:25:37.158846 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:37.158817 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vl828" Sep 19 12:25:37.728254 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:37.728228 2585 generic.go:356] "Generic (PLEG): container finished" podID="9691988f-8bb1-4962-9649-8baac63318bf" containerID="0ba9ed265b66b82bd8425517809e8540b0ee72366a55ad07e816afb520642368" exitCode=0 Sep 19 12:25:37.728421 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:37.728309 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerDied","Data":"0ba9ed265b66b82bd8425517809e8540b0ee72366a55ad07e816afb520642368"} Sep 19 12:25:38.571985 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:38.571953 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:38.572389 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:38.571965 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:38.572389 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:38.572068 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdzch" podUID="2f15ee9d-8975-4b27-b12d-290bb41beeb9" Sep 19 12:25:38.572389 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:38.572079 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:38.572389 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:38.572158 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9s4cm" podUID="ab7572c8-43f2-4f5d-9c64-fd450e74bd2c" Sep 19 12:25:38.572389 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:38.572247 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxhb9" podUID="c7a339fa-85c4-48b2-a9b4-ee3cf90464cd" Sep 19 12:25:40.192549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.192519 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:40.192994 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.192661 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:40.192994 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.192739 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs podName:c7a339fa-85c4-48b2-a9b4-ee3cf90464cd nodeName:}" failed. No retries permitted until 2026-09-19 12:26:12.192720709 +0000 UTC m=+67.799571289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs") pod "network-metrics-daemon-xxhb9" (UID: "c7a339fa-85c4-48b2-a9b4-ee3cf90464cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 12:25:40.293520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.293492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:40.293664 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.293565 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:40.293760 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.293670 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:40.293760 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.293717 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 12:25:40.293760 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.293741 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 12:25:40.293760 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.293753 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4svkn for pod openshift-network-diagnostics/network-check-target-rdzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:40.293944 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.293754 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret podName:ab7572c8-43f2-4f5d-9c64-fd450e74bd2c nodeName:}" failed. No retries permitted until 2026-09-19 12:26:12.29373626 +0000 UTC m=+67.900586819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret") pod "global-pull-secret-syncer-9s4cm" (UID: "ab7572c8-43f2-4f5d-9c64-fd450e74bd2c") : object "kube-system"/"original-pull-secret" not registered Sep 19 12:25:40.293944 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.293815 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn podName:2f15ee9d-8975-4b27-b12d-290bb41beeb9 nodeName:}" failed. No retries permitted until 2026-09-19 12:26:12.293798754 +0000 UTC m=+67.900649317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4svkn" (UniqueName: "kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn") pod "network-check-target-rdzch" (UID: "2f15ee9d-8975-4b27-b12d-290bb41beeb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 12:25:40.359052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.358977 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-149.ec2.internal" event="NodeReady" Sep 19 12:25:40.359179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.359119 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 19 12:25:40.394914 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.394886 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56588f6dcc-zsg9v"] Sep 19 12:25:40.408210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.408190 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2k8p6"] Sep 19 12:25:40.408343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.408315 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.411542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.411522 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 19 12:25:40.411645 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.411522 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 19 12:25:40.412027 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.412011 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qgdlj\"" Sep 19 12:25:40.412224 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.412211 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 19 12:25:40.417520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.417501 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 19 12:25:40.432740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.432719 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gxnxk"] Sep 19 12:25:40.432964 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.432925 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.435610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.435593 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kdhkj\"" Sep 19 12:25:40.435706 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.435599 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 19 12:25:40.435874 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.435861 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 19 12:25:40.436210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.436199 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 19 12:25:40.452848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.452829 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56588f6dcc-zsg9v"] Sep 19 12:25:40.452934 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.452864 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gxnxk"] Sep 19 12:25:40.452934 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.452889 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2k8p6"] Sep 19 12:25:40.453004 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.452959 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.455285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.455267 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 19 12:25:40.455388 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.455369 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6jvqq\"" Sep 19 12:25:40.455453 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.455444 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 19 12:25:40.494135 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6q6\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-kube-api-access-zv6q6\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494249 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-trusted-ca\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494249 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494225 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-installation-pull-secrets\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqc8x\" (UniqueName: \"kubernetes.io/projected/0022f8b9-5089-446e-a7ef-bf5a9016d8e8-kube-api-access-bqc8x\") pod \"ingress-canary-2k8p6\" (UID: \"0022f8b9-5089-446e-a7ef-bf5a9016d8e8\") " pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.494449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494357 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-image-registry-private-configuration\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494384 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-ca-trust-extracted\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494411 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-certificates\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.494449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0022f8b9-5089-446e-a7ef-bf5a9016d8e8-cert\") pod \"ingress-canary-2k8p6\" (UID: \"0022f8b9-5089-446e-a7ef-bf5a9016d8e8\") " pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.494726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.494461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-bound-sa-token\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.571494 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.571466 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:25:40.571644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.571480 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:25:40.571644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.571605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:25:40.573984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.573964 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 19 12:25:40.574274 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.574258 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 19 12:25:40.574347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.574260 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 19 12:25:40.574401 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.574379 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n4klm\"" Sep 19 12:25:40.574624 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.574606 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g8nvf\"" Sep 19 12:25:40.574624 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.574621 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 19 12:25:40.594983 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.594965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-image-registry-private-configuration\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.594989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-ca-trust-extracted\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-certificates\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0022f8b9-5089-446e-a7ef-bf5a9016d8e8-cert\") pod \"ingress-canary-2k8p6\" (UID: \"0022f8b9-5089-446e-a7ef-bf5a9016d8e8\") " pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.595070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595039 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-bound-sa-token\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.595265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-config-volume\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.595265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595191 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6q6\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-kube-api-access-zv6q6\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-trusted-ca\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595258 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccpr\" (UniqueName: \"kubernetes.io/projected/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-kube-api-access-rccpr\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.595501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595323 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-ca-trust-extracted\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-installation-pull-secrets\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.595501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqc8x\" (UniqueName: \"kubernetes.io/projected/0022f8b9-5089-446e-a7ef-bf5a9016d8e8-kube-api-access-bqc8x\") pod \"ingress-canary-2k8p6\" (UID: \"0022f8b9-5089-446e-a7ef-bf5a9016d8e8\") " pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.595501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-tmp-dir\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.595770 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.595578 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 12:25:40.595770 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.595591 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56588f6dcc-zsg9v: secret "image-registry-tls" not found Sep 19 12:25:40.595770 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.595644 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls podName:bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a nodeName:}" failed. No retries permitted until 2026-09-19 12:25:41.095627589 +0000 UTC m=+36.702478166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls") pod "image-registry-56588f6dcc-zsg9v" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a") : secret "image-registry-tls" not found Sep 19 12:25:40.595770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.595669 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-certificates\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.598864 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.598837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-image-registry-private-configuration\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.598931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.598872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-installation-pull-secrets\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.598931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.598912 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0022f8b9-5089-446e-a7ef-bf5a9016d8e8-cert\") pod \"ingress-canary-2k8p6\" (UID: \"0022f8b9-5089-446e-a7ef-bf5a9016d8e8\") " pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.603890 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.603859 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqc8x\" (UniqueName: \"kubernetes.io/projected/0022f8b9-5089-446e-a7ef-bf5a9016d8e8-kube-api-access-bqc8x\") pod \"ingress-canary-2k8p6\" (UID: \"0022f8b9-5089-446e-a7ef-bf5a9016d8e8\") " pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:40.604517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.604492 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-bound-sa-token\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.604924 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.604908 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6q6\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-kube-api-access-zv6q6\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.609620 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.609577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-trusted-ca\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:40.695948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.695917 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.695948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.695951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-config-volume\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.696137 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.695982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccpr\" (UniqueName: \"kubernetes.io/projected/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-kube-api-access-rccpr\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.696137 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.696053 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 12:25:40.696137 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.696079 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-tmp-dir\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.696137 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:40.696103 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls podName:e1aac6fa-0d58-43a6-bc0d-b2b46877c014 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:41.196089925 +0000 UTC m=+36.802940486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls") pod "dns-default-gxnxk" (UID: "e1aac6fa-0d58-43a6-bc0d-b2b46877c014") : secret "dns-default-metrics-tls" not found Sep 19 12:25:40.696422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.696405 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-tmp-dir\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.696588 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.696560 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-config-volume\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.705859 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.705838 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccpr\" (UniqueName: \"kubernetes.io/projected/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-kube-api-access-rccpr\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:40.741879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:40.741857 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2k8p6" Sep 19 12:25:41.079108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:41.078926 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2k8p6"] Sep 19 12:25:41.100227 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:41.100204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:41.100383 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:41.100365 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 12:25:41.100456 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:41.100388 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56588f6dcc-zsg9v: secret "image-registry-tls" not found Sep 19 12:25:41.100513 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:41.100457 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls podName:bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a nodeName:}" failed. No retries permitted until 2026-09-19 12:25:42.100435168 +0000 UTC m=+37.707285744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls") pod "image-registry-56588f6dcc-zsg9v" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a") : secret "image-registry-tls" not found Sep 19 12:25:41.200960 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:41.200932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:41.201335 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:41.201081 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 12:25:41.201335 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:41.201144 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls podName:e1aac6fa-0d58-43a6-bc0d-b2b46877c014 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:42.201127227 +0000 UTC m=+37.807977784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls") pod "dns-default-gxnxk" (UID: "e1aac6fa-0d58-43a6-bc0d-b2b46877c014") : secret "dns-default-metrics-tls" not found Sep 19 12:25:42.030497 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.030466 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt"] Sep 19 12:25:42.058968 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.058940 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h"] Sep 19 12:25:42.059136 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.059068 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" Sep 19 12:25:42.061550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.061528 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gs9cp\"" Sep 19 12:25:42.081064 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.081042 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d"] Sep 19 12:25:42.081172 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.081149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.083532 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.083510 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 19 12:25:42.083639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.083513 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h5kt8\"" Sep 19 12:25:42.083930 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.083911 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.084046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.083939 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.101149 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.101132 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-n6jfv"] Sep 19 12:25:42.101283 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.101260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.103514 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.103493 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 19 12:25:42.103617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.103519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 19 12:25:42.103617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.103550 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vf6hl\"" Sep 19 12:25:42.103617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.103500 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.103864 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.103844 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.107592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.107574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:42.108077 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.107733 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 12:25:42.108077 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.107753 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56588f6dcc-zsg9v: secret "image-registry-tls" not found Sep 19 12:25:42.108077 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.107823 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls podName:bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a nodeName:}" failed. No retries permitted until 2026-09-19 12:25:44.107803815 +0000 UTC m=+39.714654371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls") pod "image-registry-56588f6dcc-zsg9v" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a") : secret "image-registry-tls" not found Sep 19 12:25:42.123233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.123150 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt"] Sep 19 12:25:42.123311 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.123257 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d"] Sep 19 12:25:42.123311 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.123275 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-n6jfv"] Sep 19 12:25:42.123311 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.123213 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.123460 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.123290 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h"] Sep 19 12:25:42.123460 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.123427 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw"] Sep 19 12:25:42.125313 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.125282 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 19 12:25:42.125411 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.125316 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mr5vj\"" Sep 19 12:25:42.125411 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.125320 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 19 12:25:42.125411 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.125378 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.125769 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.125738 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.134487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.134464 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 19 12:25:42.147158 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.147140 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw"] Sep 19 12:25:42.147255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.147187 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm"] Sep 19 12:25:42.147316 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.147257 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" Sep 19 12:25:42.153519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.153421 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.153813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.153347 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qxcvw\"" Sep 19 12:25:42.154333 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.154318 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.172293 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.172272 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf"] Sep 19 12:25:42.172413 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.172393 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.175116 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.175097 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-56677\"" Sep 19 12:25:42.175193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.175125 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 19 12:25:42.175583 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.175566 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 19 12:25:42.194260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.194240 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.194351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.194257 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr"] Sep 19 12:25:42.201500 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.201476 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 19 12:25:42.201500 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.201475 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.201884 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.201584 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rvlll\"" Sep 19 12:25:42.202764 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.202745 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 19 12:25:42.204807 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.204791 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.208889 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.208870 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.208984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.208903 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd5l\" (UniqueName: \"kubernetes.io/projected/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-kube-api-access-qwd5l\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.208984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.208944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:42.209058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.208988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdx9m\" (UniqueName: \"kubernetes.io/projected/b30d083e-f915-49ec-9e0d-d13a4bf12908-kube-api-access-bdx9m\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.209058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209027 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.209158 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.209064 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 12:25:42.209158 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209118 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/072899a6-72d1-489a-b564-01e58cb221ce-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.209158 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.209150 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls podName:e1aac6fa-0d58-43a6-bc0d-b2b46877c014 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:44.209129657 +0000 UTC m=+39.815980219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls") pod "dns-default-gxnxk" (UID: "e1aac6fa-0d58-43a6-bc0d-b2b46877c014") : secret "dns-default-metrics-tls" not found Sep 19 12:25:42.209320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.209320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl47q\" (UniqueName: \"kubernetes.io/projected/2df4f088-d847-4d9e-a28b-e3b630f16d1b-kube-api-access-tl47q\") pod \"network-check-source-7cbc45fddc-nxhlt\" (UID: \"2df4f088-d847-4d9e-a28b-e3b630f16d1b\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" Sep 19 12:25:42.209320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209262 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-tmp\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.209320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209285 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-snapshots\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.209464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/072899a6-72d1-489a-b564-01e58cb221ce-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.209464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdq8\" (UniqueName: \"kubernetes.io/projected/072899a6-72d1-489a-b564-01e58cb221ce-kube-api-access-kkdq8\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.209464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209410 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-serving-cert\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.209464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.209434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nhw\" (UniqueName: \"kubernetes.io/projected/071e3fee-b7f4-4cd1-912a-7ef92690a98c-kube-api-access-w6nhw\") pod \"volume-data-source-validator-5c4db8bdf9-s94vw\" (UID: \"071e3fee-b7f4-4cd1-912a-7ef92690a98c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" Sep 19 12:25:42.217741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.217725 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-p7fnt"] Sep 19 12:25:42.217867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.217844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.221090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.221074 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 19 12:25:42.221370 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.221350 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.221493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.221393 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.221644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.221580 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 19 12:25:42.221756 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.221736 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zpq8v\"" Sep 19 12:25:42.235726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.235707 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cfdf79c4b-4q7kb"] Sep 19 12:25:42.236778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.236757 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.239061 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.239040 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 19 12:25:42.239143 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.239051 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.239217 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.239199 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 19 12:25:42.239289 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.239266 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m4tm8\"" Sep 19 12:25:42.239560 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.239541 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.243398 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.243383 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 19 12:25:42.257863 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.257843 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm"] Sep 19 12:25:42.257962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.257876 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-p7fnt"] Sep 19 12:25:42.257962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.257892 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf"] Sep 19 12:25:42.257962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.257907 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cfdf79c4b-4q7kb"] Sep 19 12:25:42.257962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.257930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr"] Sep 19 12:25:42.258155 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.257990 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.261088 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261066 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 19 12:25:42.261207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261182 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 19 12:25:42.261319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261249 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 19 12:25:42.261319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261269 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 19 12:25:42.261319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261283 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k8hkp\"" Sep 19 12:25:42.261319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261272 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 19 12:25:42.261481 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.261443 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 19 12:25:42.310276 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-serving-cert\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.310434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310289 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd5l\" (UniqueName: \"kubernetes.io/projected/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-kube-api-access-qwd5l\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.310434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310320 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/100ef318-668d-45f0-9162-2b73c34edfcc-config\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.310434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310348 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdx9m\" (UniqueName: \"kubernetes.io/projected/b30d083e-f915-49ec-9e0d-d13a4bf12908-kube-api-access-bdx9m\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.310434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310378 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.310434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/072899a6-72d1-489a-b564-01e58cb221ce-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.310434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5qd\" (UniqueName: \"kubernetes.io/projected/725756c5-21c2-4f17-b078-221ded0ffd0b-kube-api-access-4g5qd\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310553 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhpw\" (UniqueName: \"kubernetes.io/projected/22e8a67f-f965-40a9-a3eb-33cae5f93013-kube-api-access-gjhpw\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/725756c5-21c2-4f17-b078-221ded0ffd0b-config\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-tmp\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310740 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/072899a6-72d1-489a-b564-01e58cb221ce-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310774 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdq8\" (UniqueName: \"kubernetes.io/projected/072899a6-72d1-489a-b564-01e58cb221ce-kube-api-access-kkdq8\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.310790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310793 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nhw\" (UniqueName: \"kubernetes.io/projected/071e3fee-b7f4-4cd1-912a-7ef92690a98c-kube-api-access-w6nhw\") pod \"volume-data-source-validator-5c4db8bdf9-s94vw\" (UID: \"071e3fee-b7f4-4cd1-912a-7ef92690a98c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" Sep 19 12:25:42.311106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.311106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.310964 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/072899a6-72d1-489a-b564-01e58cb221ce-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.311106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311028 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/100ef318-668d-45f0-9162-2b73c34edfcc-serving-cert\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.311106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100ef318-668d-45f0-9162-2b73c34edfcc-trusted-ca\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311114 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311156 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1170f64c-37a6-414c-9206-7dd103f5692b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311153 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-tmp\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl47q\" (UniqueName: \"kubernetes.io/projected/2df4f088-d847-4d9e-a28b-e3b630f16d1b-kube-api-access-tl47q\") pod \"network-check-source-7cbc45fddc-nxhlt\" (UID: \"2df4f088-d847-4d9e-a28b-e3b630f16d1b\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.311215 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311247 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/22e8a67f-f965-40a9-a3eb-33cae5f93013-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.311281 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls podName:b30d083e-f915-49ec-9e0d-d13a4bf12908 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:42.811264847 +0000 UTC m=+38.418115405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4w58h" (UID: "b30d083e-f915-49ec-9e0d-d13a4bf12908") : secret "samples-operator-tls" not found Sep 19 12:25:42.311295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/725756c5-21c2-4f17-b078-221ded0ffd0b-serving-cert\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.311715 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311323 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-snapshots\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.311715 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311391 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bn9s\" (UniqueName: \"kubernetes.io/projected/100ef318-668d-45f0-9162-2b73c34edfcc-kube-api-access-2bn9s\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.311715 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311613 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.311848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311769 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-snapshots\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.311938 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.311889 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.314003 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.313983 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-serving-cert\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.315944 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.315922 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/072899a6-72d1-489a-b564-01e58cb221ce-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.323883 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.323840 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl47q\" (UniqueName: \"kubernetes.io/projected/2df4f088-d847-4d9e-a28b-e3b630f16d1b-kube-api-access-tl47q\") pod \"network-check-source-7cbc45fddc-nxhlt\" (UID: \"2df4f088-d847-4d9e-a28b-e3b630f16d1b\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" Sep 19 12:25:42.324792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.324313 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nhw\" (UniqueName: \"kubernetes.io/projected/071e3fee-b7f4-4cd1-912a-7ef92690a98c-kube-api-access-w6nhw\") pod \"volume-data-source-validator-5c4db8bdf9-s94vw\" (UID: \"071e3fee-b7f4-4cd1-912a-7ef92690a98c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" Sep 19 12:25:42.324792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.324759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdx9m\" (UniqueName: \"kubernetes.io/projected/b30d083e-f915-49ec-9e0d-d13a4bf12908-kube-api-access-bdx9m\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.324792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.324771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdq8\" (UniqueName: \"kubernetes.io/projected/072899a6-72d1-489a-b564-01e58cb221ce-kube-api-access-kkdq8\") pod \"kube-storage-version-migrator-operator-55889f7b47-qrv2d\" (UID: \"072899a6-72d1-489a-b564-01e58cb221ce\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.326028 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.326009 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd5l\" (UniqueName: \"kubernetes.io/projected/5eec9ccb-6d80-4c68-82d5-1cd49377dfdf-kube-api-access-qwd5l\") pod \"insights-operator-6f9c644dd9-n6jfv\" (UID: \"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf\") " pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.369388 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.369364 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" Sep 19 12:25:42.411098 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.411080 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" Sep 19 12:25:42.412356 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.412336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/100ef318-668d-45f0-9162-2b73c34edfcc-serving-cert\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.412473 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.412370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100ef318-668d-45f0-9162-2b73c34edfcc-trusted-ca\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.412473 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.412416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1170f64c-37a6-414c-9206-7dd103f5692b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.412473 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.412451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/22e8a67f-f965-40a9-a3eb-33cae5f93013-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.412625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.412476 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/725756c5-21c2-4f17-b078-221ded0ffd0b-serving-cert\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.413181 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413123 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1170f64c-37a6-414c-9206-7dd103f5692b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.413181 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413178 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-stats-auth\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.413345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413232 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.413345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bn9s\" (UniqueName: \"kubernetes.io/projected/100ef318-668d-45f0-9162-2b73c34edfcc-kube-api-access-2bn9s\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.413345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-default-certificate\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.413345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413317 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100ef318-668d-45f0-9162-2b73c34edfcc-trusted-ca\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.413345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/100ef318-668d-45f0-9162-2b73c34edfcc-config\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.413592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413381 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.413592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.413592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413400 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/22e8a67f-f965-40a9-a3eb-33cae5f93013-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.413592 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.413484 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 12:25:42.413592 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.413550 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert podName:1170f64c-37a6-414c-9206-7dd103f5692b nodeName:}" failed. No retries permitted until 2026-09-19 12:25:42.913532514 +0000 UTC m=+38.520383074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-5ckfm" (UID: "1170f64c-37a6-414c-9206-7dd103f5692b") : secret "networking-console-plugin-cert" not found Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.413657 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.413712 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls podName:22e8a67f-f965-40a9-a3eb-33cae5f93013 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:42.913678767 +0000 UTC m=+38.520529324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bvjnf" (UID: "22e8a67f-f965-40a9-a3eb-33cae5f93013") : secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5qd\" (UniqueName: \"kubernetes.io/projected/725756c5-21c2-4f17-b078-221ded0ffd0b-kube-api-access-4g5qd\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhpw\" (UniqueName: \"kubernetes.io/projected/22e8a67f-f965-40a9-a3eb-33cae5f93013-kube-api-access-gjhpw\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413866 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/725756c5-21c2-4f17-b078-221ded0ffd0b-config\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413917 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9dfq\" (UniqueName: \"kubernetes.io/projected/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-kube-api-access-w9dfq\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.414131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.413918 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/100ef318-668d-45f0-9162-2b73c34edfcc-config\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.414423 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.414249 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/725756c5-21c2-4f17-b078-221ded0ffd0b-config\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.415229 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.415212 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/725756c5-21c2-4f17-b078-221ded0ffd0b-serving-cert\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.415361 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.415342 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/100ef318-668d-45f0-9162-2b73c34edfcc-serving-cert\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.422338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.422321 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhpw\" (UniqueName: \"kubernetes.io/projected/22e8a67f-f965-40a9-a3eb-33cae5f93013-kube-api-access-gjhpw\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.422338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.422329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5qd\" (UniqueName: \"kubernetes.io/projected/725756c5-21c2-4f17-b078-221ded0ffd0b-kube-api-access-4g5qd\") pod \"service-ca-operator-57cdc4b768-s66kr\" (UID: \"725756c5-21c2-4f17-b078-221ded0ffd0b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.422533 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.422510 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bn9s\" (UniqueName: \"kubernetes.io/projected/100ef318-668d-45f0-9162-2b73c34edfcc-kube-api-access-2bn9s\") pod \"console-operator-fd4bb46c-p7fnt\" (UID: \"100ef318-668d-45f0-9162-2b73c34edfcc\") " pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.436238 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.436220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" Sep 19 12:25:42.456630 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.456608 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" Sep 19 12:25:42.515282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.515255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-stats-auth\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.515400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.515307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.515400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.515352 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-default-certificate\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.515506 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.515407 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.515506 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.515465 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9dfq\" (UniqueName: \"kubernetes.io/projected/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-kube-api-access-w9dfq\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.515759 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.515679 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 12:25:42.515759 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.515753 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:43.01573105 +0000 UTC m=+38.622581627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : configmap references non-existent config key: service-ca.crt Sep 19 12:25:42.515924 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.515784 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:43.015771332 +0000 UTC m=+38.622621889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : secret "router-metrics-certs-default" not found Sep 19 12:25:42.518255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.518216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-stats-auth\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.518394 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.518376 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-default-certificate\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.526022 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.526000 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9dfq\" (UniqueName: \"kubernetes.io/projected/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-kube-api-access-w9dfq\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:42.527337 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.527323 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" Sep 19 12:25:42.547194 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.547174 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:42.596327 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.596306 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0022f8b9_5089_446e_a7ef_bf5a9016d8e8.slice/crio-88b1a611f8b89fc8b6891ce85b25c8c11d43b4a3fa6defc2fa32c06f1873b445 WatchSource:0}: Error finding container 88b1a611f8b89fc8b6891ce85b25c8c11d43b4a3fa6defc2fa32c06f1873b445: Status 404 returned error can't find the container with id 88b1a611f8b89fc8b6891ce85b25c8c11d43b4a3fa6defc2fa32c06f1873b445 Sep 19 12:25:42.741231 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.741200 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2k8p6" event={"ID":"0022f8b9-5089-446e-a7ef-bf5a9016d8e8","Type":"ContainerStarted","Data":"88b1a611f8b89fc8b6891ce85b25c8c11d43b4a3fa6defc2fa32c06f1873b445"} Sep 19 12:25:42.818096 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.818036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:42.818211 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.818193 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 12:25:42.818277 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.818264 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls podName:b30d083e-f915-49ec-9e0d-d13a4bf12908 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:43.818248581 +0000 UTC m=+39.425099138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4w58h" (UID: "b30d083e-f915-49ec-9e0d-d13a4bf12908") : secret "samples-operator-tls" not found Sep 19 12:25:42.904509 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.904480 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw"] Sep 19 12:25:42.916519 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.916496 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071e3fee_b7f4_4cd1_912a_7ef92690a98c.slice/crio-7042340e795b367a393596fae5f12d17e43ed0e228cf2200078585def1c743da WatchSource:0}: Error finding container 7042340e795b367a393596fae5f12d17e43ed0e228cf2200078585def1c743da: Status 404 returned error can't find the container with id 7042340e795b367a393596fae5f12d17e43ed0e228cf2200078585def1c743da Sep 19 12:25:42.919625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.919596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:42.919750 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.919646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:42.919840 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.919813 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:42.919927 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.919880 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 12:25:42.919927 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.919884 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls podName:22e8a67f-f965-40a9-a3eb-33cae5f93013 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:43.919863514 +0000 UTC m=+39.526714091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bvjnf" (UID: "22e8a67f-f965-40a9-a3eb-33cae5f93013") : secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:42.920015 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:42.919934 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert podName:1170f64c-37a6-414c-9206-7dd103f5692b nodeName:}" failed. No retries permitted until 2026-09-19 12:25:43.919920442 +0000 UTC m=+39.526771002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-5ckfm" (UID: "1170f64c-37a6-414c-9206-7dd103f5692b") : secret "networking-console-plugin-cert" not found Sep 19 12:25:42.925799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.925781 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-p7fnt"] Sep 19 12:25:42.928371 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.928350 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100ef318_668d_45f0_9162_2b73c34edfcc.slice/crio-e377e370183ba1127628dbc0f59370f210ea15f8d47686f4f7b30aa86a69820c WatchSource:0}: Error finding container e377e370183ba1127628dbc0f59370f210ea15f8d47686f4f7b30aa86a69820c: Status 404 returned error can't find the container with id e377e370183ba1127628dbc0f59370f210ea15f8d47686f4f7b30aa86a69820c Sep 19 12:25:42.934375 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.934354 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt"] Sep 19 12:25:42.935793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.935763 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr"] Sep 19 12:25:42.936713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.936673 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d"] Sep 19 12:25:42.937542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:42.937526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-n6jfv"] Sep 19 12:25:42.938827 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.938774 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725756c5_21c2_4f17_b078_221ded0ffd0b.slice/crio-64701154d7e478b8f9606baf3152e2c19eff3a35b3f9bd44239ce9948f6cfb52 WatchSource:0}: Error finding container 64701154d7e478b8f9606baf3152e2c19eff3a35b3f9bd44239ce9948f6cfb52: Status 404 returned error can't find the container with id 64701154d7e478b8f9606baf3152e2c19eff3a35b3f9bd44239ce9948f6cfb52 Sep 19 12:25:42.940297 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.939756 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eec9ccb_6d80_4c68_82d5_1cd49377dfdf.slice/crio-00c3de24dc19cc300eacb3cf6a08595d7988ef0af2370a39c4d6df95b84a7887 WatchSource:0}: Error finding container 00c3de24dc19cc300eacb3cf6a08595d7988ef0af2370a39c4d6df95b84a7887: Status 404 returned error can't find the container with id 00c3de24dc19cc300eacb3cf6a08595d7988ef0af2370a39c4d6df95b84a7887 Sep 19 12:25:42.940792 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.940771 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072899a6_72d1_489a_b564_01e58cb221ce.slice/crio-83455d2311f4294286b3bc7b41faef8282a2fb7b04df481d60c001230589c7b2 WatchSource:0}: Error finding container 83455d2311f4294286b3bc7b41faef8282a2fb7b04df481d60c001230589c7b2: Status 404 returned error can't find the container with id 83455d2311f4294286b3bc7b41faef8282a2fb7b04df481d60c001230589c7b2 Sep 19 12:25:42.942026 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:25:42.941557 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df4f088_d847_4d9e_a28b_e3b630f16d1b.slice/crio-89ffe52d03c52f4b605b0ef468b7c3f1ca259d772a7779c1c6c5faefe5c43473 WatchSource:0}: Error finding container 89ffe52d03c52f4b605b0ef468b7c3f1ca259d772a7779c1c6c5faefe5c43473: Status 404 returned error can't find the container with id 89ffe52d03c52f4b605b0ef468b7c3f1ca259d772a7779c1c6c5faefe5c43473 Sep 19 12:25:43.020154 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.020130 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:43.020258 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.020239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:43.020304 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.020269 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 12:25:43.020347 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.020336 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:44.020321738 +0000 UTC m=+39.627172300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : secret "router-metrics-certs-default" not found Sep 19 12:25:43.020413 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.020401 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:44.020384416 +0000 UTC m=+39.627234978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : configmap references non-existent config key: service-ca.crt Sep 19 12:25:43.748739 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.748500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" event={"ID":"072899a6-72d1-489a-b564-01e58cb221ce","Type":"ContainerStarted","Data":"83455d2311f4294286b3bc7b41faef8282a2fb7b04df481d60c001230589c7b2"} Sep 19 12:25:43.750374 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.750343 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" event={"ID":"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf","Type":"ContainerStarted","Data":"00c3de24dc19cc300eacb3cf6a08595d7988ef0af2370a39c4d6df95b84a7887"} Sep 19 12:25:43.756120 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.756046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" event={"ID":"2df4f088-d847-4d9e-a28b-e3b630f16d1b","Type":"ContainerStarted","Data":"89ffe52d03c52f4b605b0ef468b7c3f1ca259d772a7779c1c6c5faefe5c43473"} Sep 19 12:25:43.759176 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.759105 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" event={"ID":"071e3fee-b7f4-4cd1-912a-7ef92690a98c","Type":"ContainerStarted","Data":"7042340e795b367a393596fae5f12d17e43ed0e228cf2200078585def1c743da"} Sep 19 12:25:43.765321 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.764373 2585 generic.go:356] "Generic (PLEG): container finished" podID="9691988f-8bb1-4962-9649-8baac63318bf" containerID="d716dbc67129aece5062cf56fc7acfa42624d64b25168599e0f02e8f82b4a04d" exitCode=0 Sep 19 12:25:43.765321 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.764448 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerDied","Data":"d716dbc67129aece5062cf56fc7acfa42624d64b25168599e0f02e8f82b4a04d"} Sep 19 12:25:43.768835 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.768757 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" event={"ID":"725756c5-21c2-4f17-b078-221ded0ffd0b","Type":"ContainerStarted","Data":"64701154d7e478b8f9606baf3152e2c19eff3a35b3f9bd44239ce9948f6cfb52"} Sep 19 12:25:43.772947 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.772891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" event={"ID":"100ef318-668d-45f0-9162-2b73c34edfcc","Type":"ContainerStarted","Data":"e377e370183ba1127628dbc0f59370f210ea15f8d47686f4f7b30aa86a69820c"} Sep 19 12:25:43.830066 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.829418 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:43.830066 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.829645 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 12:25:43.830066 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.829726 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls podName:b30d083e-f915-49ec-9e0d-d13a4bf12908 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:45.829706758 +0000 UTC m=+41.436557328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4w58h" (UID: "b30d083e-f915-49ec-9e0d-d13a4bf12908") : secret "samples-operator-tls" not found Sep 19 12:25:43.931720 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.930393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:43.931720 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:43.930440 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:43.931720 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.931184 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 12:25:43.931720 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.931643 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:43.932504 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.932387 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert podName:1170f64c-37a6-414c-9206-7dd103f5692b nodeName:}" failed. No retries permitted until 2026-09-19 12:25:45.932366768 +0000 UTC m=+41.539217340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-5ckfm" (UID: "1170f64c-37a6-414c-9206-7dd103f5692b") : secret "networking-console-plugin-cert" not found Sep 19 12:25:43.932504 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:43.932413 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls podName:22e8a67f-f965-40a9-a3eb-33cae5f93013 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:45.932402521 +0000 UTC m=+41.539253104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bvjnf" (UID: "22e8a67f-f965-40a9-a3eb-33cae5f93013") : secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:44.033536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:44.033507 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:44.033640 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:44.033601 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:44.033778 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.033763 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 12:25:44.033837 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.033827 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:46.033808797 +0000 UTC m=+41.640659357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : secret "router-metrics-certs-default" not found Sep 19 12:25:44.034016 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.033970 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:46.033957943 +0000 UTC m=+41.640808521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : configmap references non-existent config key: service-ca.crt Sep 19 12:25:44.134373 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:44.134341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:44.134601 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.134585 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 12:25:44.134676 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.134605 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56588f6dcc-zsg9v: secret "image-registry-tls" not found Sep 19 12:25:44.134676 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.134661 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls podName:bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a nodeName:}" failed. No retries permitted until 2026-09-19 12:25:48.134643859 +0000 UTC m=+43.741494420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls") pod "image-registry-56588f6dcc-zsg9v" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a") : secret "image-registry-tls" not found Sep 19 12:25:44.235804 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:44.235212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:44.235804 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.235404 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 12:25:44.235804 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:44.235464 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls podName:e1aac6fa-0d58-43a6-bc0d-b2b46877c014 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:48.235447059 +0000 UTC m=+43.842297620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls") pod "dns-default-gxnxk" (UID: "e1aac6fa-0d58-43a6-bc0d-b2b46877c014") : secret "dns-default-metrics-tls" not found Sep 19 12:25:44.807458 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:44.806901 2585 generic.go:356] "Generic (PLEG): container finished" podID="9691988f-8bb1-4962-9649-8baac63318bf" containerID="9365be99f3fc2c615dfa3b86f602169918f79d4653ea54d55a976ad8ca9fa395" exitCode=0 Sep 19 12:25:44.808566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:44.808154 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerDied","Data":"9365be99f3fc2c615dfa3b86f602169918f79d4653ea54d55a976ad8ca9fa395"} Sep 19 12:25:45.815438 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:45.815379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" event={"ID":"9691988f-8bb1-4962-9649-8baac63318bf","Type":"ContainerStarted","Data":"b97fe61f2c8cc37b15c0abdd3e766c203b2cb06431618a73ebe62a2747e68419"} Sep 19 12:25:45.838245 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:45.838192 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jj7nj" podStartSLOduration=9.741411416 podStartE2EDuration="39.838173502s" totalImagesPullingTime="30.096762086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:45.836611526 +0000 UTC m=+41.443462132" watchObservedRunningTime="2026-09-19 12:25:45.838173502 +0000 UTC m=+41.445024081" Sep 19 12:25:45.857849 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:45.857820 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:45.858021 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:45.858006 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 12:25:45.858086 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:45.858068 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls podName:b30d083e-f915-49ec-9e0d-d13a4bf12908 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:49.858049009 +0000 UTC m=+45.464899566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4w58h" (UID: "b30d083e-f915-49ec-9e0d-d13a4bf12908") : secret "samples-operator-tls" not found Sep 19 12:25:45.958917 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:45.958881 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:45.959091 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:45.958929 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:45.959091 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:45.958998 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 12:25:45.959091 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:45.959009 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:45.959091 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:45.959086 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert podName:1170f64c-37a6-414c-9206-7dd103f5692b nodeName:}" failed. No retries permitted until 2026-09-19 12:25:49.959064488 +0000 UTC m=+45.565915059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-5ckfm" (UID: "1170f64c-37a6-414c-9206-7dd103f5692b") : secret "networking-console-plugin-cert" not found Sep 19 12:25:45.959337 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:45.959153 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls podName:22e8a67f-f965-40a9-a3eb-33cae5f93013 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:49.959131857 +0000 UTC m=+45.565982435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bvjnf" (UID: "22e8a67f-f965-40a9-a3eb-33cae5f93013") : secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:46.059392 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:46.059361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:46.059527 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:46.059399 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:50.059384474 +0000 UTC m=+45.666235051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : configmap references non-existent config key: service-ca.crt Sep 19 12:25:46.059527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:46.059459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:46.059663 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:46.059641 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 12:25:46.059755 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:46.059743 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:50.059723738 +0000 UTC m=+45.666574298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : secret "router-metrics-certs-default" not found Sep 19 12:25:48.178246 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:48.178209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:48.178745 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:48.178402 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 12:25:48.178745 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:48.178426 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56588f6dcc-zsg9v: secret "image-registry-tls" not found Sep 19 12:25:48.178745 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:48.178493 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls podName:bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a nodeName:}" failed. No retries permitted until 2026-09-19 12:25:56.178471568 +0000 UTC m=+51.785322125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls") pod "image-registry-56588f6dcc-zsg9v" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a") : secret "image-registry-tls" not found Sep 19 12:25:48.279474 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:48.279441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:48.279628 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:48.279582 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 12:25:48.279738 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:48.279644 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls podName:e1aac6fa-0d58-43a6-bc0d-b2b46877c014 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:56.279626626 +0000 UTC m=+51.886477182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls") pod "dns-default-gxnxk" (UID: "e1aac6fa-0d58-43a6-bc0d-b2b46877c014") : secret "dns-default-metrics-tls" not found Sep 19 12:25:49.893182 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:49.893144 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:49.893611 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:49.893316 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 12:25:49.893611 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:49.893386 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls podName:b30d083e-f915-49ec-9e0d-d13a4bf12908 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:57.893363023 +0000 UTC m=+53.500213579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4w58h" (UID: "b30d083e-f915-49ec-9e0d-d13a4bf12908") : secret "samples-operator-tls" not found Sep 19 12:25:49.994165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:49.994129 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:49.994343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:49.994173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:49.994343 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:49.994275 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 12:25:49.994343 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:49.994337 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:49.994506 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:49.994340 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert podName:1170f64c-37a6-414c-9206-7dd103f5692b nodeName:}" failed. No retries permitted until 2026-09-19 12:25:57.994324249 +0000 UTC m=+53.601174809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-5ckfm" (UID: "1170f64c-37a6-414c-9206-7dd103f5692b") : secret "networking-console-plugin-cert" not found Sep 19 12:25:49.994506 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:49.994403 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls podName:22e8a67f-f965-40a9-a3eb-33cae5f93013 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:57.994384806 +0000 UTC m=+53.601235374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bvjnf" (UID: "22e8a67f-f965-40a9-a3eb-33cae5f93013") : secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:50.095192 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.095159 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:50.095321 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.095225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:50.095321 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:50.095313 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:58.095295334 +0000 UTC m=+53.702145890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : configmap references non-existent config key: service-ca.crt Sep 19 12:25:50.095408 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:50.095387 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 12:25:50.095453 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:50.095442 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:25:58.095427634 +0000 UTC m=+53.702278191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : secret "router-metrics-certs-default" not found Sep 19 12:25:50.856476 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.856334 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" event={"ID":"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf","Type":"ContainerStarted","Data":"e6e4f32f200d40a89493b6008236b0adb8dfaebafdb56a1a5aa27f9b0007afdc"} Sep 19 12:25:50.858240 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.858195 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" event={"ID":"2df4f088-d847-4d9e-a28b-e3b630f16d1b","Type":"ContainerStarted","Data":"cccde9d41482cb275981b83fd6f7232945a15d25711500a89276e932ca7a0882"} Sep 19 12:25:50.860655 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.860631 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" event={"ID":"071e3fee-b7f4-4cd1-912a-7ef92690a98c","Type":"ContainerStarted","Data":"8ae6fb8bf9ac0e654e3b2b0934417f827a1dab830ca8c813024070a2e5b014ce"} Sep 19 12:25:50.864702 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.862675 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2k8p6" event={"ID":"0022f8b9-5089-446e-a7ef-bf5a9016d8e8","Type":"ContainerStarted","Data":"a5526ef343973f6692f68881b4aac72e2619f62736b71f24c53fb1e32db91fee"} Sep 19 12:25:50.864985 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.864733 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" event={"ID":"725756c5-21c2-4f17-b078-221ded0ffd0b","Type":"ContainerStarted","Data":"429b585aed59a83c4c51872c0831d9270eb3f5a202b37a6ac9832e155a040d61"} Sep 19 12:25:50.867032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.867011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" event={"ID":"100ef318-668d-45f0-9162-2b73c34edfcc","Type":"ContainerStarted","Data":"1dfd6b4afc509874bcbd8faa8273a2ffaac4480395a124eb137c9975f1e2a1d8"} Sep 19 12:25:50.869098 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.869081 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:50.870479 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.870445 2585 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-p7fnt container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Sep 19 12:25:50.870636 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.870606 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Sep 19 12:25:50.872316 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.872297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" event={"ID":"072899a6-72d1-489a-b564-01e58cb221ce","Type":"ContainerStarted","Data":"29aa3759a69c52a1908c6ae72b28e3b0f94c963bed17b3278156ffe4c3d5336c"} Sep 19 12:25:50.899971 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.899909 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" podStartSLOduration=1.235018049 podStartE2EDuration="8.899892066s" totalImagesPullingTime="7.664874017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:50.899408143 +0000 UTC m=+46.506258748" watchObservedRunningTime="2026-09-19 12:25:50.899892066 +0000 UTC m=+46.506742645" Sep 19 12:25:50.929582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.929249 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nxhlt" podStartSLOduration=1.25510059 podStartE2EDuration="8.929230118s" totalImagesPullingTime="7.674129528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:50.928832052 +0000 UTC m=+46.535682656" watchObservedRunningTime="2026-09-19 12:25:50.929230118 +0000 UTC m=+46.536080693" Sep 19 12:25:50.971267 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:50.971213 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" podStartSLOduration=1.306139643 podStartE2EDuration="8.971194659s" totalImagesPullingTime="7.665055016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:50.970378372 +0000 UTC m=+46.577228978" watchObservedRunningTime="2026-09-19 12:25:50.971194659 +0000 UTC m=+46.578045239" Sep 19 12:25:51.031814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.031349 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s94vw" podStartSLOduration=1.393938364 podStartE2EDuration="9.031328941s" totalImagesPullingTime="7.637390577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:51.030839954 +0000 UTC m=+46.637690526" watchObservedRunningTime="2026-09-19 12:25:51.031328941 +0000 UTC m=+46.638179522" Sep 19 12:25:51.033181 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.032932 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2k8p6" podStartSLOduration=3.024980751 podStartE2EDuration="11.032903477s" totalImagesPullingTime="8.007922726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:51.00812922 +0000 UTC m=+46.614979825" watchObservedRunningTime="2026-09-19 12:25:51.032903477 +0000 UTC m=+46.639754057" Sep 19 12:25:51.078143 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.078091 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podStartSLOduration=1.402224259 podStartE2EDuration="9.078076475s" totalImagesPullingTime="7.675852216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:51.052463412 +0000 UTC m=+46.659314009" watchObservedRunningTime="2026-09-19 12:25:51.078076475 +0000 UTC m=+46.684927054" Sep 19 12:25:51.078373 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.078350 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" podStartSLOduration=1.413378065 podStartE2EDuration="9.078340524s" totalImagesPullingTime="7.664962459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:25:51.077877079 +0000 UTC m=+46.684727685" watchObservedRunningTime="2026-09-19 12:25:51.078340524 +0000 UTC m=+46.685191097" Sep 19 12:25:51.878355 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.878324 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/0.log" Sep 19 12:25:51.878526 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.878369 2585 generic.go:356] "Generic (PLEG): container finished" podID="100ef318-668d-45f0-9162-2b73c34edfcc" containerID="1dfd6b4afc509874bcbd8faa8273a2ffaac4480395a124eb137c9975f1e2a1d8" exitCode=255 Sep 19 12:25:51.878526 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.878472 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" event={"ID":"100ef318-668d-45f0-9162-2b73c34edfcc","Type":"ContainerDied","Data":"1dfd6b4afc509874bcbd8faa8273a2ffaac4480395a124eb137c9975f1e2a1d8"} Sep 19 12:25:51.878761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:51.878745 2585 scope.go:117] "RemoveContainer" containerID="1dfd6b4afc509874bcbd8faa8273a2ffaac4480395a124eb137c9975f1e2a1d8" Sep 19 12:25:52.547644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.547611 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:25:52.612796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.612747 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d7hjq"] Sep 19 12:25:52.614864 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.614850 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.617134 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.617114 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 19 12:25:52.617327 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.617304 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 19 12:25:52.617409 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.617369 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-765zc\"" Sep 19 12:25:52.626720 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.626700 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7hjq"] Sep 19 12:25:52.718439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.718398 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.718616 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.718489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.718616 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.718563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-crio-socket\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.718616 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.718596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-data-volume\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.718822 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.718650 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb262\" (UniqueName: \"kubernetes.io/projected/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-api-access-zb262\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.819727 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.819635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-crio-socket\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.819727 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.819711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-data-volume\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.819912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.819762 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb262\" (UniqueName: \"kubernetes.io/projected/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-api-access-zb262\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.819912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.819789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.819912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.819876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.819912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.819884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-crio-socket\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.820100 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:52.819950 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 12:25:52.820100 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:52.819988 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 12:25:52.820100 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:52.820043 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:25:53.320020562 +0000 UTC m=+48.926871135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : configmap "kube-rbac-proxy" not found Sep 19 12:25:52.820100 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:52.820074 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:25:53.320063986 +0000 UTC m=+48.926914547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : secret "insights-runtime-extractor-tls" not found Sep 19 12:25:52.820100 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.820042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-data-volume\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.857415 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.857390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb262\" (UniqueName: \"kubernetes.io/projected/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-api-access-zb262\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:52.882921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.882900 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/1.log" Sep 19 12:25:52.883332 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.883316 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/0.log" Sep 19 12:25:52.883396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.883349 2585 generic.go:356] "Generic (PLEG): container finished" podID="100ef318-668d-45f0-9162-2b73c34edfcc" containerID="3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef" exitCode=255 Sep 19 12:25:52.883396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.883391 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" event={"ID":"100ef318-668d-45f0-9162-2b73c34edfcc","Type":"ContainerDied","Data":"3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef"} Sep 19 12:25:52.883464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.883428 2585 scope.go:117] "RemoveContainer" containerID="1dfd6b4afc509874bcbd8faa8273a2ffaac4480395a124eb137c9975f1e2a1d8" Sep 19 12:25:52.883720 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:52.883672 2585 scope.go:117] "RemoveContainer" containerID="3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef" Sep 19 12:25:52.883909 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:52.883885 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-p7fnt_openshift-console-operator(100ef318-668d-45f0-9162-2b73c34edfcc)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" Sep 19 12:25:53.324007 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:53.323977 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:53.324136 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:53.324034 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:53.324136 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:53.324096 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 12:25:53.324136 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:53.324112 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 12:25:53.324249 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:53.324157 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:25:54.324141587 +0000 UTC m=+49.930992144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : configmap "kube-rbac-proxy" not found Sep 19 12:25:53.324249 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:53.324173 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:25:54.324166108 +0000 UTC m=+49.931016664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : secret "insights-runtime-extractor-tls" not found Sep 19 12:25:53.888104 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:53.888083 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/1.log" Sep 19 12:25:53.888444 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:53.888424 2585 scope.go:117] "RemoveContainer" containerID="3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef" Sep 19 12:25:53.888614 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:53.888594 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-p7fnt_openshift-console-operator(100ef318-668d-45f0-9162-2b73c34edfcc)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" Sep 19 12:25:54.332056 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:54.332020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:54.332228 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:54.332087 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:54.332228 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:54.332131 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 12:25:54.332228 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:54.332206 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:25:56.332186006 +0000 UTC m=+51.939036580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : configmap "kube-rbac-proxy" not found Sep 19 12:25:54.332399 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:54.332227 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 12:25:54.332399 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:54.332285 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:25:56.332270827 +0000 UTC m=+51.939121393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : secret "insights-runtime-extractor-tls" not found Sep 19 12:25:54.383186 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:54.383166 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2nn4d_d62b4d7a-0062-4806-ba22-6fc3dc29e3d2/dns-node-resolver/0.log" Sep 19 12:25:55.383527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:55.383504 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5b7xp_7d3187f2-8b12-441d-bc4f-17bcaa9b12e9/node-ca/0.log" Sep 19 12:25:56.183402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:56.183375 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2k8p6_0022f8b9-5089-446e-a7ef-bf5a9016d8e8/serve-healthcheck-canary/0.log" Sep 19 12:25:56.248250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:56.248222 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:25:56.248392 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.248375 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 12:25:56.248455 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.248396 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56588f6dcc-zsg9v: secret "image-registry-tls" not found Sep 19 12:25:56.248508 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.248461 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls podName:bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a nodeName:}" failed. No retries permitted until 2026-09-19 12:26:12.248441805 +0000 UTC m=+67.855292368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls") pod "image-registry-56588f6dcc-zsg9v" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a") : secret "image-registry-tls" not found Sep 19 12:25:56.349387 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:56.349359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:25:56.349509 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:56.349406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:56.349509 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.349475 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 12:25:56.349620 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.349512 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:26:00.349501323 +0000 UTC m=+55.956351880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : configmap "kube-rbac-proxy" not found Sep 19 12:25:56.349620 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.349519 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 12:25:56.349620 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:56.349540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:25:56.349620 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.349616 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls podName:e1aac6fa-0d58-43a6-bc0d-b2b46877c014 nodeName:}" failed. No retries permitted until 2026-09-19 12:26:12.349609185 +0000 UTC m=+67.956459741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls") pod "dns-default-gxnxk" (UID: "e1aac6fa-0d58-43a6-bc0d-b2b46877c014") : secret "dns-default-metrics-tls" not found Sep 19 12:25:56.349822 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.349623 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 12:25:56.349822 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:56.349663 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:26:00.349647449 +0000 UTC m=+55.956498023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : secret "insights-runtime-extractor-tls" not found Sep 19 12:25:57.960395 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:57.960365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:25:57.960736 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:57.960507 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 12:25:57.960736 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:57.960561 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls podName:b30d083e-f915-49ec-9e0d-d13a4bf12908 nodeName:}" failed. No retries permitted until 2026-09-19 12:26:13.960547102 +0000 UTC m=+69.567397659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4w58h" (UID: "b30d083e-f915-49ec-9e0d-d13a4bf12908") : secret "samples-operator-tls" not found Sep 19 12:25:58.061589 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:58.061560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:25:58.061724 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:58.061594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:25:58.061767 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.061753 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:58.061801 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.061767 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 12:25:58.061839 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.061811 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls podName:22e8a67f-f965-40a9-a3eb-33cae5f93013 nodeName:}" failed. No retries permitted until 2026-09-19 12:26:14.061798668 +0000 UTC m=+69.668649224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bvjnf" (UID: "22e8a67f-f965-40a9-a3eb-33cae5f93013") : secret "cluster-monitoring-operator-tls" not found Sep 19 12:25:58.061839 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.061830 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert podName:1170f64c-37a6-414c-9206-7dd103f5692b nodeName:}" failed. No retries permitted until 2026-09-19 12:26:14.061817759 +0000 UTC m=+69.668668316 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-5ckfm" (UID: "1170f64c-37a6-414c-9206-7dd103f5692b") : secret "networking-console-plugin-cert" not found Sep 19 12:25:58.162296 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:58.162268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:58.162411 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:25:58.162340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:25:58.162411 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.162374 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:26:14.162359673 +0000 UTC m=+69.769210230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : configmap references non-existent config key: service-ca.crt Sep 19 12:25:58.162497 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.162419 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 12:25:58.162497 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:25:58.162471 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs podName:cb7ad890-3bbd-408c-b9f2-8975c8dd5542 nodeName:}" failed. No retries permitted until 2026-09-19 12:26:14.162454119 +0000 UTC m=+69.769304688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs") pod "router-default-5cfdf79c4b-4q7kb" (UID: "cb7ad890-3bbd-408c-b9f2-8975c8dd5542") : secret "router-metrics-certs-default" not found Sep 19 12:26:00.380792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:00.380757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:26:00.381179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:00.380822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:26:00.381179 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:00.380953 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 12:26:00.381179 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:00.381007 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls podName:4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d nodeName:}" failed. No retries permitted until 2026-09-19 12:26:08.38099334 +0000 UTC m=+63.987843901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7hjq" (UID: "4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d") : secret "insights-runtime-extractor-tls" not found Sep 19 12:26:00.381378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:00.381357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:26:00.869247 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:00.869222 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:26:00.869566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:00.869553 2585 scope.go:117] "RemoveContainer" containerID="3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef" Sep 19 12:26:00.869726 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:00.869708 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-p7fnt_openshift-console-operator(100ef318-668d-45f0-9162-2b73c34edfcc)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" Sep 19 12:26:02.548065 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.548032 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:26:02.548636 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.548494 2585 scope.go:117] "RemoveContainer" containerID="3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef" Sep 19 12:26:02.917215 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.917157 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:26:02.917570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.917557 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/1.log" Sep 19 12:26:02.917616 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.917591 2585 generic.go:356] "Generic (PLEG): container finished" podID="100ef318-668d-45f0-9162-2b73c34edfcc" containerID="dbf8954d18f72f4b4f24c8f4a436fcdadd3f546346341861c535993e522f2356" exitCode=255 Sep 19 12:26:02.917650 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.917634 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" event={"ID":"100ef318-668d-45f0-9162-2b73c34edfcc","Type":"ContainerDied","Data":"dbf8954d18f72f4b4f24c8f4a436fcdadd3f546346341861c535993e522f2356"} Sep 19 12:26:02.917714 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.917701 2585 scope.go:117] "RemoveContainer" containerID="3643ccffdc9a118f4b46c1a8ad3b0884a7889cdf97a37e8a1d00960f78e48eef" Sep 19 12:26:02.917980 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:02.917961 2585 scope.go:117] "RemoveContainer" containerID="dbf8954d18f72f4b4f24c8f4a436fcdadd3f546346341861c535993e522f2356" Sep 19 12:26:02.918161 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:02.918140 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-p7fnt_openshift-console-operator(100ef318-668d-45f0-9162-2b73c34edfcc)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" Sep 19 12:26:03.925575 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:03.925546 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:26:06.739358 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:06.739334 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjpc" Sep 19 12:26:08.451233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.451202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:26:08.453582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.453561 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hjq\" (UID: \"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d\") " pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:26:08.526977 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.526955 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-765zc\"" Sep 19 12:26:08.534865 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.534846 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7hjq" Sep 19 12:26:08.808044 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.808019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7hjq"] Sep 19 12:26:08.811116 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:08.811091 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe736dd_2ae1_4c4b_bd4e_a09b2fdfbb0d.slice/crio-e45d4da807760a5c7a442cbc27e1d886362c5141d295044d2a782078df52970e WatchSource:0}: Error finding container e45d4da807760a5c7a442cbc27e1d886362c5141d295044d2a782078df52970e: Status 404 returned error can't find the container with id e45d4da807760a5c7a442cbc27e1d886362c5141d295044d2a782078df52970e Sep 19 12:26:08.940895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.940869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hjq" event={"ID":"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d","Type":"ContainerStarted","Data":"4db71911ba3f8d13dd2151ee16238cf76d9eea95fdc9215435decdad233ec319"} Sep 19 12:26:08.941009 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:08.940919 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hjq" event={"ID":"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d","Type":"ContainerStarted","Data":"e45d4da807760a5c7a442cbc27e1d886362c5141d295044d2a782078df52970e"} Sep 19 12:26:09.945316 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:09.945246 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hjq" event={"ID":"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d","Type":"ContainerStarted","Data":"a065cf60926fd3d3c783399f43f3bc06178aa9a4f600bffbf00200fea299505f"} Sep 19 12:26:10.869434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:10.869400 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:26:10.869822 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:10.869801 2585 scope.go:117] "RemoveContainer" containerID="dbf8954d18f72f4b4f24c8f4a436fcdadd3f546346341861c535993e522f2356" Sep 19 12:26:10.869985 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:10.869966 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-p7fnt_openshift-console-operator(100ef318-668d-45f0-9162-2b73c34edfcc)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" Sep 19 12:26:11.953400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:11.953367 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hjq" event={"ID":"4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d","Type":"ContainerStarted","Data":"8f74b94e67ea9f21ff5151b56ff5655af05c5f8fca592562df1f281aba320b33"} Sep 19 12:26:11.971391 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:11.971347 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d7hjq" podStartSLOduration=17.832039319 podStartE2EDuration="19.971332822s" totalImagesPullingTime="2.139293503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:11.970899152 +0000 UTC m=+67.577749736" watchObservedRunningTime="2026-09-19 12:26:11.971332822 +0000 UTC m=+67.578183400" Sep 19 12:26:12.282353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.282259 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:26:12.282353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.282310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:26:12.284841 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.284816 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"image-registry-56588f6dcc-zsg9v\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:26:12.284939 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.284876 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 19 12:26:12.294637 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.294611 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a339fa-85c4-48b2-a9b4-ee3cf90464cd-metrics-certs\") pod \"network-metrics-daemon-xxhb9\" (UID: \"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd\") " pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:26:12.383222 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.383199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:26:12.383338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.383229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:26:12.383338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.383306 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:26:12.383454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.383375 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g8nvf\"" Sep 19 12:26:12.385384 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.385360 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 19 12:26:12.385605 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.385587 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1aac6fa-0d58-43a6-bc0d-b2b46877c014-metrics-tls\") pod \"dns-default-gxnxk\" (UID: \"e1aac6fa-0d58-43a6-bc0d-b2b46877c014\") " pod="openshift-dns/dns-default-gxnxk" Sep 19 12:26:12.385672 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.385644 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svkn\" (UniqueName: \"kubernetes.io/projected/2f15ee9d-8975-4b27-b12d-290bb41beeb9-kube-api-access-4svkn\") pod \"network-check-target-rdzch\" (UID: \"2f15ee9d-8975-4b27-b12d-290bb41beeb9\") " pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:26:12.392212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.392197 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxhb9" Sep 19 12:26:12.395763 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.395746 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab7572c8-43f2-4f5d-9c64-fd450e74bd2c-original-pull-secret\") pod \"global-pull-secret-syncer-9s4cm\" (UID: \"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c\") " pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:26:12.396292 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.396276 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n4klm\"" Sep 19 12:26:12.404254 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.404236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:26:12.521263 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.521241 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qgdlj\"" Sep 19 12:26:12.529387 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.529371 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:26:12.547781 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.547763 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:26:12.548104 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.548088 2585 scope.go:117] "RemoveContainer" containerID="dbf8954d18f72f4b4f24c8f4a436fcdadd3f546346341861c535993e522f2356" Sep 19 12:26:12.548276 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:12.548255 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-p7fnt_openshift-console-operator(100ef318-668d-45f0-9162-2b73c34edfcc)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" podUID="100ef318-668d-45f0-9162-2b73c34edfcc" Sep 19 12:26:12.565360 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.565339 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6jvqq\"" Sep 19 12:26:12.573662 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.573551 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxnxk" Sep 19 12:26:12.666931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.666901 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xxhb9"] Sep 19 12:26:12.669182 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:12.669153 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a339fa_85c4_48b2_a9b4_ee3cf90464cd.slice/crio-2aabff94d34d70c5882a7893d5de2f36c0183914784b75d294923ad3edecf2d2 WatchSource:0}: Error finding container 2aabff94d34d70c5882a7893d5de2f36c0183914784b75d294923ad3edecf2d2: Status 404 returned error can't find the container with id 2aabff94d34d70c5882a7893d5de2f36c0183914784b75d294923ad3edecf2d2 Sep 19 12:26:12.687410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.687387 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9s4cm" Sep 19 12:26:12.693116 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.693098 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rdzch"] Sep 19 12:26:12.695295 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:12.695273 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f15ee9d_8975_4b27_b12d_290bb41beeb9.slice/crio-155c26f9846b38daff47f0e6e20fffbcd3889f50a26aef18dd673c7aef01d1e2 WatchSource:0}: Error finding container 155c26f9846b38daff47f0e6e20fffbcd3889f50a26aef18dd673c7aef01d1e2: Status 404 returned error can't find the container with id 155c26f9846b38daff47f0e6e20fffbcd3889f50a26aef18dd673c7aef01d1e2 Sep 19 12:26:12.800246 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.800222 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56588f6dcc-zsg9v"] Sep 19 12:26:12.802154 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:12.802129 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9bf34a_d0e5_4ad8_936f_f417a8a50c2a.slice/crio-726aa292af2c80f91b190a5f7df73143ad697d52a288ec67c92ce3523dd9b8fe WatchSource:0}: Error finding container 726aa292af2c80f91b190a5f7df73143ad697d52a288ec67c92ce3523dd9b8fe: Status 404 returned error can't find the container with id 726aa292af2c80f91b190a5f7df73143ad697d52a288ec67c92ce3523dd9b8fe Sep 19 12:26:12.850059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.850040 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gxnxk"] Sep 19 12:26:12.852211 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:12.852188 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1aac6fa_0d58_43a6_bc0d_b2b46877c014.slice/crio-3ae5a7a2291b0465afb96faf9eaa570114ee780c8964b6656d375a0ccf9e10da WatchSource:0}: Error finding container 3ae5a7a2291b0465afb96faf9eaa570114ee780c8964b6656d375a0ccf9e10da: Status 404 returned error can't find the container with id 3ae5a7a2291b0465afb96faf9eaa570114ee780c8964b6656d375a0ccf9e10da Sep 19 12:26:12.957893 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.957856 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9s4cm"] Sep 19 12:26:12.959278 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.959242 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rdzch" event={"ID":"2f15ee9d-8975-4b27-b12d-290bb41beeb9","Type":"ContainerStarted","Data":"4e369dcdf28258ce77ed5b3383d26b323f50d5f10b3a2245127a77496d686cb0"} Sep 19 12:26:12.959385 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.959304 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rdzch" event={"ID":"2f15ee9d-8975-4b27-b12d-290bb41beeb9","Type":"ContainerStarted","Data":"155c26f9846b38daff47f0e6e20fffbcd3889f50a26aef18dd673c7aef01d1e2"} Sep 19 12:26:12.959385 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.959333 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:26:12.960611 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:12.960587 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7572c8_43f2_4f5d_9c64_fd450e74bd2c.slice/crio-5b8bd54b0aaee7ff7cbb3a3efd59fcc78c5cfd47b579bcf5ec106cf103bd8899 WatchSource:0}: Error finding container 5b8bd54b0aaee7ff7cbb3a3efd59fcc78c5cfd47b579bcf5ec106cf103bd8899: Status 404 returned error can't find the container with id 5b8bd54b0aaee7ff7cbb3a3efd59fcc78c5cfd47b579bcf5ec106cf103bd8899 Sep 19 12:26:12.960733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.960704 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxhb9" event={"ID":"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd","Type":"ContainerStarted","Data":"2aabff94d34d70c5882a7893d5de2f36c0183914784b75d294923ad3edecf2d2"} Sep 19 12:26:12.962442 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.962418 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" event={"ID":"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a","Type":"ContainerStarted","Data":"3fa17e048c6a3eff3044e9691c77a7bd3e6b9f6ff681322ff2b9f6144a97fa35"} Sep 19 12:26:12.962536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.962460 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" event={"ID":"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a","Type":"ContainerStarted","Data":"726aa292af2c80f91b190a5f7df73143ad697d52a288ec67c92ce3523dd9b8fe"} Sep 19 12:26:12.962536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.962486 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:26:12.963710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:12.963663 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxnxk" event={"ID":"e1aac6fa-0d58-43a6-bc0d-b2b46877c014","Type":"ContainerStarted","Data":"3ae5a7a2291b0465afb96faf9eaa570114ee780c8964b6656d375a0ccf9e10da"} Sep 19 12:26:13.035813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:13.035761 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rdzch" podStartSLOduration=67.035743597 podStartE2EDuration="1m7.035743597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:13.008928399 +0000 UTC m=+68.615778997" watchObservedRunningTime="2026-09-19 12:26:13.035743597 +0000 UTC m=+68.642594176" Sep 19 12:26:13.036134 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:13.036109 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" podStartSLOduration=66.036103129 podStartE2EDuration="1m6.036103129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:13.035896749 +0000 UTC m=+68.642747321" watchObservedRunningTime="2026-09-19 12:26:13.036103129 +0000 UTC m=+68.642953708" Sep 19 12:26:13.970496 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:13.970464 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9s4cm" event={"ID":"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c","Type":"ContainerStarted","Data":"5b8bd54b0aaee7ff7cbb3a3efd59fcc78c5cfd47b579bcf5ec106cf103bd8899"} Sep 19 12:26:14.000095 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:13.998499 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:26:14.002400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.002340 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b30d083e-f915-49ec-9e0d-d13a4bf12908-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4w58h\" (UID: \"b30d083e-f915-49ec-9e0d-d13a4bf12908\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:26:14.099871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.099622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:26:14.099871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.099665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:26:14.103079 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.103054 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1170f64c-37a6-414c-9206-7dd103f5692b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-5ckfm\" (UID: \"1170f64c-37a6-414c-9206-7dd103f5692b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:26:14.103816 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.103792 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/22e8a67f-f965-40a9-a3eb-33cae5f93013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bvjnf\" (UID: \"22e8a67f-f965-40a9-a3eb-33cae5f93013\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:26:14.194160 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.194093 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h5kt8\"" Sep 19 12:26:14.200461 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.200432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:14.200591 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.200514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:14.201206 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.201179 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-service-ca-bundle\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:14.201923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.201904 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" Sep 19 12:26:14.203140 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.203121 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7ad890-3bbd-408c-b9f2-8975c8dd5542-metrics-certs\") pod \"router-default-5cfdf79c4b-4q7kb\" (UID: \"cb7ad890-3bbd-408c-b9f2-8975c8dd5542\") " pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:14.286223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.286198 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-56677\"" Sep 19 12:26:14.294453 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.294424 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" Sep 19 12:26:14.306058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.306036 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rvlll\"" Sep 19 12:26:14.314066 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.314048 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" Sep 19 12:26:14.369920 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.369900 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k8hkp\"" Sep 19 12:26:14.378011 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.377984 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:14.862068 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.861855 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf"] Sep 19 12:26:14.867950 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:14.867927 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e8a67f_f965_40a9_a3eb_33cae5f93013.slice/crio-47f4665ad8a6f94aa7666669527d4ac42ce65dbdaf07ac5e62d12a8eaae0bb40 WatchSource:0}: Error finding container 47f4665ad8a6f94aa7666669527d4ac42ce65dbdaf07ac5e62d12a8eaae0bb40: Status 404 returned error can't find the container with id 47f4665ad8a6f94aa7666669527d4ac42ce65dbdaf07ac5e62d12a8eaae0bb40 Sep 19 12:26:14.910296 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.910197 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h"] Sep 19 12:26:14.920543 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.920516 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cfdf79c4b-4q7kb"] Sep 19 12:26:14.923367 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.923316 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm"] Sep 19 12:26:14.927011 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:14.926953 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7ad890_3bbd_408c_b9f2_8975c8dd5542.slice/crio-2b3ae4fd9842449d12f72ef9150ab1e874da5d16b57023b3d4dc57fb8ab554d1 WatchSource:0}: Error finding container 2b3ae4fd9842449d12f72ef9150ab1e874da5d16b57023b3d4dc57fb8ab554d1: Status 404 returned error can't find the container with id 2b3ae4fd9842449d12f72ef9150ab1e874da5d16b57023b3d4dc57fb8ab554d1 Sep 19 12:26:14.927206 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:14.927180 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1170f64c_37a6_414c_9206_7dd103f5692b.slice/crio-68fe4bc38b6c057fdce11cdf530584c7c1e5ea8f4e7fe346f628d797e58e3476 WatchSource:0}: Error finding container 68fe4bc38b6c057fdce11cdf530584c7c1e5ea8f4e7fe346f628d797e58e3476: Status 404 returned error can't find the container with id 68fe4bc38b6c057fdce11cdf530584c7c1e5ea8f4e7fe346f628d797e58e3476 Sep 19 12:26:14.981453 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.981417 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" event={"ID":"cb7ad890-3bbd-408c-b9f2-8975c8dd5542","Type":"ContainerStarted","Data":"2b3ae4fd9842449d12f72ef9150ab1e874da5d16b57023b3d4dc57fb8ab554d1"} Sep 19 12:26:14.983959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.983930 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxnxk" event={"ID":"e1aac6fa-0d58-43a6-bc0d-b2b46877c014","Type":"ContainerStarted","Data":"05d4e459f93a2d820214db9757122d91e55997ab8e1d6985786537bc8ec749ee"} Sep 19 12:26:14.984215 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.984195 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxnxk" event={"ID":"e1aac6fa-0d58-43a6-bc0d-b2b46877c014","Type":"ContainerStarted","Data":"b0a526c302c5b97a1026c052d7a9694d3d099d378d303631fbcb689f4dd3dafc"} Sep 19 12:26:14.984295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.984235 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gxnxk" Sep 19 12:26:14.988437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.987899 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxhb9" event={"ID":"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd","Type":"ContainerStarted","Data":"52d37908c364b4c83e67b93ae630f9773a9049942a0f3f1d3e3c23a91b9b0d18"} Sep 19 12:26:14.988437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.987945 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxhb9" event={"ID":"c7a339fa-85c4-48b2-a9b4-ee3cf90464cd","Type":"ContainerStarted","Data":"d8c64650d4ab366970cd3052703d86b9becbd0f5be73602c5a3ce041d2173bf5"} Sep 19 12:26:14.989719 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.989663 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" event={"ID":"1170f64c-37a6-414c-9206-7dd103f5692b","Type":"ContainerStarted","Data":"68fe4bc38b6c057fdce11cdf530584c7c1e5ea8f4e7fe346f628d797e58e3476"} Sep 19 12:26:14.991494 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.991452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" event={"ID":"22e8a67f-f965-40a9-a3eb-33cae5f93013","Type":"ContainerStarted","Data":"47f4665ad8a6f94aa7666669527d4ac42ce65dbdaf07ac5e62d12a8eaae0bb40"} Sep 19 12:26:14.992883 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:14.992840 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" event={"ID":"b30d083e-f915-49ec-9e0d-d13a4bf12908","Type":"ContainerStarted","Data":"0d9a51c81e9138260a2cdf9d1a91e7cb3e205e0784190a61769c72288c7e2367"} Sep 19 12:26:15.002614 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:15.002578 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gxnxk" podStartSLOduration=33.227917409 podStartE2EDuration="35.002565548s" totalImagesPullingTime="1.774648139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:15.000320897 +0000 UTC m=+70.607171495" watchObservedRunningTime="2026-09-19 12:26:15.002565548 +0000 UTC m=+70.609416126" Sep 19 12:26:16.001922 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:16.001787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" event={"ID":"cb7ad890-3bbd-408c-b9f2-8975c8dd5542","Type":"ContainerStarted","Data":"82063015a4bae304c7215955e3a909d5a50ef43c29bf9bd7a35604dcf45f55cb"} Sep 19 12:26:16.024456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:16.023065 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xxhb9" podStartSLOduration=68.82462168 podStartE2EDuration="1m10.023044308s" totalImagesPullingTime="1.198422628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:15.01696729 +0000 UTC m=+70.623817890" watchObservedRunningTime="2026-09-19 12:26:16.023044308 +0000 UTC m=+71.629894893" Sep 19 12:26:16.024456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:16.023409 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" podStartSLOduration=34.023398695 podStartE2EDuration="34.023398695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:16.022956751 +0000 UTC m=+71.629807331" watchObservedRunningTime="2026-09-19 12:26:16.023398695 +0000 UTC m=+71.630249278" Sep 19 12:26:16.379078 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:16.379043 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:16.382260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:16.382051 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:17.006705 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:17.006402 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:17.007653 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:17.007628 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cfdf79c4b-4q7kb" Sep 19 12:26:19.530765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.530733 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c"] Sep 19 12:26:19.563324 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.563299 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c"] Sep 19 12:26:19.563446 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.563405 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:19.566065 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.566043 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lcgjs\"" Sep 19 12:26:19.566201 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.566042 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 19 12:26:19.645141 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.645121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d44a0101-d10b-4f9a-b861-3de9ab9ef6dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qvf4c\" (UID: \"d44a0101-d10b-4f9a-b861-3de9ab9ef6dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:19.746397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:19.746369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d44a0101-d10b-4f9a-b861-3de9ab9ef6dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qvf4c\" (UID: \"d44a0101-d10b-4f9a-b861-3de9ab9ef6dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:19.746496 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:19.746474 2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 19 12:26:19.746535 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:19.746525 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44a0101-d10b-4f9a-b861-3de9ab9ef6dd-tls-certificates podName:d44a0101-d10b-4f9a-b861-3de9ab9ef6dd nodeName:}" failed. No retries permitted until 2026-09-19 12:26:20.246510667 +0000 UTC m=+75.853361223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d44a0101-d10b-4f9a-b861-3de9ab9ef6dd-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" (UID: "d44a0101-d10b-4f9a-b861-3de9ab9ef6dd") : secret "prometheus-operator-admission-webhook-tls" not found Sep 19 12:26:20.021311 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.021285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" event={"ID":"22e8a67f-f965-40a9-a3eb-33cae5f93013","Type":"ContainerStarted","Data":"e8b92a96e23586fd7a5bbdcac151e17fe16457836dac216f090f6e084fb90156"} Sep 19 12:26:20.023158 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.023132 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" event={"ID":"b30d083e-f915-49ec-9e0d-d13a4bf12908","Type":"ContainerStarted","Data":"5e03792025cbdddfc215773b503a96957b5ed28d36c00aa4707ef169b79d3b13"} Sep 19 12:26:20.023282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.023186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" event={"ID":"b30d083e-f915-49ec-9e0d-d13a4bf12908","Type":"ContainerStarted","Data":"1f7430eeb09abcc808aedd2365210edda9bc458e46c7531949f8bebaaa3286dd"} Sep 19 12:26:20.024764 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.024741 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9s4cm" event={"ID":"ab7572c8-43f2-4f5d-9c64-fd450e74bd2c","Type":"ContainerStarted","Data":"236df76157bcfdbc1b3a44fe020ffa839b635fad23626cd9113e0f636ff1f587"} Sep 19 12:26:20.026147 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.026120 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" event={"ID":"1170f64c-37a6-414c-9206-7dd103f5692b","Type":"ContainerStarted","Data":"c13b058670f0d07d5890d88a1da890ac4986da93f0f6f5240b8c363be3875085"} Sep 19 12:26:20.043590 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.043548 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bvjnf" podStartSLOduration=33.974622556 podStartE2EDuration="38.043512791s" totalImagesPullingTime="4.068890235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:20.043023233 +0000 UTC m=+75.649873833" watchObservedRunningTime="2026-09-19 12:26:20.043512791 +0000 UTC m=+75.650363371" Sep 19 12:26:20.061954 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.061917 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4w58h" podStartSLOduration=34.086481129 podStartE2EDuration="38.061903262s" totalImagesPullingTime="3.975422133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:20.060475855 +0000 UTC m=+75.667326456" watchObservedRunningTime="2026-09-19 12:26:20.061903262 +0000 UTC m=+75.668753840" Sep 19 12:26:20.079892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.079849 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9s4cm" podStartSLOduration=68.107286558 podStartE2EDuration="1m14.079837159s" totalImagesPullingTime="5.972550601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:20.078728696 +0000 UTC m=+75.685579277" watchObservedRunningTime="2026-09-19 12:26:20.079837159 +0000 UTC m=+75.686687777" Sep 19 12:26:20.095103 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.095064 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-5ckfm" podStartSLOduration=34.088929654 podStartE2EDuration="38.095055763s" totalImagesPullingTime="4.006126109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:20.094402616 +0000 UTC m=+75.701253206" watchObservedRunningTime="2026-09-19 12:26:20.095055763 +0000 UTC m=+75.701906342" Sep 19 12:26:20.251171 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.251135 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d44a0101-d10b-4f9a-b861-3de9ab9ef6dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qvf4c\" (UID: \"d44a0101-d10b-4f9a-b861-3de9ab9ef6dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:20.253605 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.253586 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d44a0101-d10b-4f9a-b861-3de9ab9ef6dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qvf4c\" (UID: \"d44a0101-d10b-4f9a-b861-3de9ab9ef6dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:20.471869 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.471843 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:20.760142 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:20.760068 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c"] Sep 19 12:26:20.762908 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:20.762877 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44a0101_d10b_4f9a_b861_3de9ab9ef6dd.slice/crio-68626d0925bcd8c6e136112db00c0d00b109a67331beb7a683d7de210f9765fc WatchSource:0}: Error finding container 68626d0925bcd8c6e136112db00c0d00b109a67331beb7a683d7de210f9765fc: Status 404 returned error can't find the container with id 68626d0925bcd8c6e136112db00c0d00b109a67331beb7a683d7de210f9765fc Sep 19 12:26:21.030369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:21.030308 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" event={"ID":"d44a0101-d10b-4f9a-b861-3de9ab9ef6dd","Type":"ContainerStarted","Data":"68626d0925bcd8c6e136112db00c0d00b109a67331beb7a683d7de210f9765fc"} Sep 19 12:26:23.039457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:23.039425 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" event={"ID":"d44a0101-d10b-4f9a-b861-3de9ab9ef6dd","Type":"ContainerStarted","Data":"c6f4f4b372049e9c93138e4f33cdb54349d1a967658cc98000490d4794d7f05a"} Sep 19 12:26:23.039896 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:23.039703 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:23.044424 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:23.044396 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" Sep 19 12:26:23.056508 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:23.056467 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qvf4c" podStartSLOduration=2.818083614 podStartE2EDuration="4.056452786s" totalImagesPullingTime="1.238369172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:23.05584033 +0000 UTC m=+78.662690929" watchObservedRunningTime="2026-09-19 12:26:23.056452786 +0000 UTC m=+78.663303365" Sep 19 12:26:24.572670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:24.572640 2585 scope.go:117] "RemoveContainer" containerID="dbf8954d18f72f4b4f24c8f4a436fcdadd3f546346341861c535993e522f2356" Sep 19 12:26:25.049396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:25.049363 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:26:25.049560 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:25.049494 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" event={"ID":"100ef318-668d-45f0-9162-2b73c34edfcc","Type":"ContainerStarted","Data":"bc0a7285bbbc71f6cc5cbdb7653040263b1a6f672e16a2fcf854a8aad23a2521"} Sep 19 12:26:25.049839 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:25.049810 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:26:25.808951 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:25.808925 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-p7fnt" Sep 19 12:26:26.008145 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:26.008119 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gxnxk" Sep 19 12:26:27.971545 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.971509 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5w5c7"] Sep 19 12:26:27.974412 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.974395 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:27.980710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.977591 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 19 12:26:27.980710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.977962 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f82qw\"" Sep 19 12:26:27.980710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.978963 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 19 12:26:27.980710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.979296 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 19 12:26:27.980710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.979506 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 19 12:26:27.992466 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.992439 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6x9f9"] Sep 19 12:26:27.994639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.994624 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:27.998547 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.998525 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 19 12:26:27.998846 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.998830 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hcrxn\"" Sep 19 12:26:27.998932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.998867 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 19 12:26:27.999472 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:27.999456 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 19 12:26:28.018084 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.018062 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6x9f9"] Sep 19 12:26:28.108921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.108883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02dae659-fd13-4d8c-ab31-e7663a42af3e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.109082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.108927 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-wtmp\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.108964 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-textfile\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.108996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-root\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-metrics-client-ca\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109046 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109304 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109083 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.109304 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-sys\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109372 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.109372 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109363 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-accelerators-collector-config\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109379 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-tls\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.109454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109421 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/02dae659-fd13-4d8c-ab31-e7663a42af3e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.109454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109436 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.109454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109450 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pql2c\" (UniqueName: \"kubernetes.io/projected/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-api-access-pql2c\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.109589 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.109469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5mj\" (UniqueName: \"kubernetes.io/projected/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-kube-api-access-ff5mj\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210389 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-textfile\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210389 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-root\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-metrics-client-ca\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210426 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210448 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210466 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-sys\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210479 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-root\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210522 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-sys\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210546 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-accelerators-collector-config\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-tls\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/02dae659-fd13-4d8c-ab31-e7663a42af3e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210755 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pql2c\" (UniqueName: \"kubernetes.io/projected/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-api-access-pql2c\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210803 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-textfile\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5mj\" (UniqueName: \"kubernetes.io/projected/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-kube-api-access-ff5mj\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210864 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02dae659-fd13-4d8c-ab31-e7663a42af3e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.210966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.210890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-wtmp\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.211540 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.211065 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-wtmp\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.211540 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.211166 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-metrics-client-ca\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.211667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.211643 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/02dae659-fd13-4d8c-ab31-e7663a42af3e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.211757 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.211722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.212053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.211941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-accelerators-collector-config\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.212348 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.212324 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02dae659-fd13-4d8c-ab31-e7663a42af3e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.213434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.213411 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.213517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.213468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.213517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.213473 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-node-exporter-tls\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.213633 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.213601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.220989 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.220964 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pql2c\" (UniqueName: \"kubernetes.io/projected/02dae659-fd13-4d8c-ab31-e7663a42af3e-kube-api-access-pql2c\") pod \"kube-state-metrics-84c885658c-6x9f9\" (UID: \"02dae659-fd13-4d8c-ab31-e7663a42af3e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.221152 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.221124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5mj\" (UniqueName: \"kubernetes.io/projected/f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51-kube-api-access-ff5mj\") pod \"node-exporter-5w5c7\" (UID: \"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51\") " pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.287176 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.287158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5w5c7" Sep 19 12:26:28.297292 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:28.297265 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421ea6a_8ee3_4c1f_a3cd_a1b904d20f51.slice/crio-7ad8150cc511dba5916f26ee2b906484c0fd3dbd132d8447befdb972fc17f14d WatchSource:0}: Error finding container 7ad8150cc511dba5916f26ee2b906484c0fd3dbd132d8447befdb972fc17f14d: Status 404 returned error can't find the container with id 7ad8150cc511dba5916f26ee2b906484c0fd3dbd132d8447befdb972fc17f14d Sep 19 12:26:28.302858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.302839 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" Sep 19 12:26:28.578139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:28.578118 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6x9f9"] Sep 19 12:26:28.579751 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:28.579729 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dae659_fd13_4d8c_ab31_e7663a42af3e.slice/crio-a4e2583b108f55ca0654986d0bcf880c3e55a3606c8d24c275a0327782e508f5 WatchSource:0}: Error finding container a4e2583b108f55ca0654986d0bcf880c3e55a3606c8d24c275a0327782e508f5: Status 404 returned error can't find the container with id a4e2583b108f55ca0654986d0bcf880c3e55a3606c8d24c275a0327782e508f5 Sep 19 12:26:29.069091 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.069040 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w5c7" event={"ID":"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51","Type":"ContainerStarted","Data":"7ad8150cc511dba5916f26ee2b906484c0fd3dbd132d8447befdb972fc17f14d"} Sep 19 12:26:29.070925 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.070896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" event={"ID":"02dae659-fd13-4d8c-ab31-e7663a42af3e","Type":"ContainerStarted","Data":"a4e2583b108f55ca0654986d0bcf880c3e55a3606c8d24c275a0327782e508f5"} Sep 19 12:26:29.118602 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.118565 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:26:29.121263 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.121241 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.123826 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.123804 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 19 12:26:29.123969 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.123933 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 19 12:26:29.124019 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.123961 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 19 12:26:29.124019 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.123934 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 19 12:26:29.124142 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.123969 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xcqw4\"" Sep 19 12:26:29.124223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.124204 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 19 12:26:29.124358 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.124337 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 19 12:26:29.124434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.124346 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 19 12:26:29.124434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.124411 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 19 12:26:29.124504 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.124466 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 19 12:26:29.139268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.139244 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:26:29.220009 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.219975 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-out\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220024 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220086 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220120 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220154 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v8pm\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-kube-api-access-9v8pm\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220204 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220240 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220292 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-volume\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220307 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220331 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-web-config\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.220788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.220435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.321623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321543 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-web-config\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.321623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.321877 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.321877 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-out\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.321877 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.321877 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321884 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321907 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v8pm\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-kube-api-access-9v8pm\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.321996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.322022 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.322046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-volume\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.322048 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.322068 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.322413 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:29.322158 2585 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 19 12:26:29.322413 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:26:29.322229 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls podName:056eea10-a03c-4c26-9b8d-4ed8cfcb930e nodeName:}" failed. No retries permitted until 2026-09-19 12:26:29.822207041 +0000 UTC m=+85.429057611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e") : secret "alertmanager-main-tls" not found Sep 19 12:26:29.323048 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.323001 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.324176 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.324101 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.326274 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.325621 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-out\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.328041 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.328012 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.329641 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.329618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-volume\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.330352 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.330306 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-web-config\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.330451 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.330382 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.330451 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.330417 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.330548 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.330460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.333011 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.332973 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.334219 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.334177 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v8pm\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-kube-api-access-9v8pm\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.826928 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.826887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:29.829937 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:29.829909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:30.031833 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.031807 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57d495d4cb-f69b5"] Sep 19 12:26:30.034661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.034632 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:26:30.037044 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.037024 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.040446 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.040375 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 19 12:26:30.040861 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.040829 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5mwtv\"" Sep 19 12:26:30.041195 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.041175 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 19 12:26:30.044122 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.041389 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 19 12:26:30.044223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.041448 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 19 12:26:30.044283 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.041496 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7hn3oce5e9jq2\"" Sep 19 12:26:30.044339 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.041584 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 19 12:26:30.050108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.049870 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57d495d4cb-f69b5"] Sep 19 12:26:30.079499 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.079448 2585 generic.go:356] "Generic (PLEG): container finished" podID="f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51" containerID="c9c5dbe448086609a112ec969b6bc7bbfc8607b7aeadb77954695dbd49cf685f" exitCode=0 Sep 19 12:26:30.079923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.079637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w5c7" event={"ID":"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51","Type":"ContainerDied","Data":"c9c5dbe448086609a112ec969b6bc7bbfc8607b7aeadb77954695dbd49cf685f"} Sep 19 12:26:30.082870 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.082838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" event={"ID":"02dae659-fd13-4d8c-ab31-e7663a42af3e","Type":"ContainerStarted","Data":"e61e5e5ae7ad1191df005141cf6d00a21698760290dc6049082e5778d8afb0f7"} Sep 19 12:26:30.082966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.082907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" event={"ID":"02dae659-fd13-4d8c-ab31-e7663a42af3e","Type":"ContainerStarted","Data":"c0ea26f881283aad72929e15f34d535f940059195d4c51c53ebabbd08312037c"} Sep 19 12:26:30.123168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.123105 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" podStartSLOduration=1.816737076 podStartE2EDuration="3.123088081s" totalImagesPullingTime="1.306351005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:30.121763122 +0000 UTC m=+85.728613729" watchObservedRunningTime="2026-09-19 12:26:30.123088081 +0000 UTC m=+85.729938688" Sep 19 12:26:30.130026 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130004 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-tls\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130120 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130238 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-metrics-client-ca\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130238 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130194 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-grpc-tls\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130238 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn74\" (UniqueName: \"kubernetes.io/projected/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-kube-api-access-pxn74\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.130349 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.130319 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.230934 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.230909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231156 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.230942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231156 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.231101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231279 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.231170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-metrics-client-ca\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231279 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.231210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-grpc-tls\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231383 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.231317 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn74\" (UniqueName: \"kubernetes.io/projected/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-kube-api-access-pxn74\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231444 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.231378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.231544 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.231449 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-tls\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.233565 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.232530 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-metrics-client-ca\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.234618 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.234149 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.234618 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.234147 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.234618 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.234579 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.234832 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.234717 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-grpc-tls\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.235658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.235636 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.236085 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.236070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-secret-thanos-querier-tls\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.241032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.241014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn74\" (UniqueName: \"kubernetes.io/projected/bbcaf35f-2a69-4c0a-8b72-c0496aa00e80-kube-api-access-pxn74\") pod \"thanos-querier-57d495d4cb-f69b5\" (UID: \"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80\") " pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.336631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.336598 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:26:30.338960 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:30.338919 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056eea10_a03c_4c26_9b8d_4ed8cfcb930e.slice/crio-4171873762d3aa7df1a99005ea697b0c5ff8d629a67dcf65dbba98c5b72472f3 WatchSource:0}: Error finding container 4171873762d3aa7df1a99005ea697b0c5ff8d629a67dcf65dbba98c5b72472f3: Status 404 returned error can't find the container with id 4171873762d3aa7df1a99005ea697b0c5ff8d629a67dcf65dbba98c5b72472f3 Sep 19 12:26:30.363171 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.363134 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:30.641639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:30.641568 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57d495d4cb-f69b5"] Sep 19 12:26:30.645284 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:30.645258 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcaf35f_2a69_4c0a_8b72_c0496aa00e80.slice/crio-2d7454c004f2591461a3ec312f743270ee1f8742f26ab327a5f80c2e3c096e88 WatchSource:0}: Error finding container 2d7454c004f2591461a3ec312f743270ee1f8742f26ab327a5f80c2e3c096e88: Status 404 returned error can't find the container with id 2d7454c004f2591461a3ec312f743270ee1f8742f26ab327a5f80c2e3c096e88 Sep 19 12:26:31.090532 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:31.090496 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w5c7" event={"ID":"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51","Type":"ContainerStarted","Data":"7e1092c8d53d73a441acffcc96c162a3effe4555e47deefb6b90e426a3abd8e0"} Sep 19 12:26:31.091040 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:31.090563 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w5c7" event={"ID":"f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51","Type":"ContainerStarted","Data":"5cf8266611c70dbf9c4c5d21d2a100f15607a887c0c715233e0dce3c830bcae3"} Sep 19 12:26:31.091928 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:31.091904 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"2d7454c004f2591461a3ec312f743270ee1f8742f26ab327a5f80c2e3c096e88"} Sep 19 12:26:31.093452 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:31.093415 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"4171873762d3aa7df1a99005ea697b0c5ff8d629a67dcf65dbba98c5b72472f3"} Sep 19 12:26:31.095804 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:31.095745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6x9f9" event={"ID":"02dae659-fd13-4d8c-ab31-e7663a42af3e","Type":"ContainerStarted","Data":"aa2ad67922710f882faa2a5fa25f04176a1b4645f752cb2b2bced18b188866f6"} Sep 19 12:26:31.112721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:31.112655 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5w5c7" podStartSLOduration=2.9455979020000003 podStartE2EDuration="4.11263682s" totalImagesPullingTime="1.091674192s" totalInitContainerRuntime="75.364726ms" isStatefulPod=true podCreationTimestamp="2026-09-19 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:31.11161052 +0000 UTC m=+86.718461122" watchObservedRunningTime="2026-09-19 12:26:31.11263682 +0000 UTC m=+86.719487403" Sep 19 12:26:32.101435 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:32.101392 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="48c7d80b5a6b52e6f7af1771f2acbbab5f1fb53f10f43080bd9e0c17e0cae25c" exitCode=0 Sep 19 12:26:32.101876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:32.101479 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"48c7d80b5a6b52e6f7af1771f2acbbab5f1fb53f10f43080bd9e0c17e0cae25c"} Sep 19 12:26:33.108172 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:33.108139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"979aa5cc7dd70e96f8bdc71a8d33cd34ea410112493f1d33544df169a4e227c2"} Sep 19 12:26:33.108527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:33.108201 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"d799c3698b3fc14a052260c8da93dd2f108055a99593f0ff78154340bdc92bf6"} Sep 19 12:26:33.108527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:33.108226 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"c0bd95c9ce295163e61a9e8b3ca8bf5c3fc028c5e6e2e402767f8e92561d6d50"} Sep 19 12:26:33.975812 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:33.975788 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:26:34.124997 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.124971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"88554e4771896767f2ade46e430c59083bbd1a25902c0c336937cf0ba58a5cd6"} Sep 19 12:26:34.125359 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.125054 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"b89fae70fed4f964c67b346b322f78d7d65027017ff3483f8d98e2bc38727847"} Sep 19 12:26:34.125359 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.125081 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"1bbc8c9abf6f492b73fdf9635954b02a605597f4387fd27e6b4c377e88e4b616"} Sep 19 12:26:34.129292 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.129270 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"22e9b343e47528eca9ba03a75a57cc1638f20cab118addb66f541123f0a25c51"} Sep 19 12:26:34.129460 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.129439 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"0287a26429062542c8c0a06f195f05f3e604341d9323a45e3cce2aa49d0ae4bb"} Sep 19 12:26:34.129575 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.129473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" event={"ID":"bbcaf35f-2a69-4c0a-8b72-c0496aa00e80","Type":"ContainerStarted","Data":"d419273fa917c28f1942ed7b4adc27f17d40de2da68813ffb5befca941ede998"} Sep 19 12:26:34.129787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.129771 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:34.152185 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:34.152148 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" podStartSLOduration=1.099671992 podStartE2EDuration="4.152132705s" totalImagesPullingTime="3.052460713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:26:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:34.151025259 +0000 UTC m=+89.757875872" watchObservedRunningTime="2026-09-19 12:26:34.152132705 +0000 UTC m=+89.758983284" Sep 19 12:26:35.137754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:35.137720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"0152d3cf616a148bcbb04c41f2fcd0bbd5594916780f92130b19c2c66fea06b3"} Sep 19 12:26:35.138152 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:35.137789 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"48c593c90c189f387e490999b5a6ad085fd9507be77accfa318a7f7dbcc26c57"} Sep 19 12:26:35.138152 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:35.137821 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerStarted","Data":"a9ec795db45d1f9b97e53b80c8ada9a0f8d72e5b979e57b78049ba6a2c736429"} Sep 19 12:26:35.168770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:35.168712 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.227282173 podStartE2EDuration="6.168670093s" totalImagesPullingTime="2.922099342s" totalInitContainerRuntime="19.288578ms" isStatefulPod=false podCreationTimestamp="2026-09-19 12:26:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:35.165905996 +0000 UTC m=+90.772756606" watchObservedRunningTime="2026-09-19 12:26:35.168670093 +0000 UTC m=+90.775520672" Sep 19 12:26:39.035578 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:39.035536 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56588f6dcc-zsg9v"] Sep 19 12:26:40.145787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:40.145759 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57d495d4cb-f69b5" Sep 19 12:26:43.973511 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:43.973480 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rdzch" Sep 19 12:26:46.906545 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.906508 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c88d59dd5-tmxrc"] Sep 19 12:26:46.909993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.909973 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:46.913850 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.913825 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 19 12:26:46.914040 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.914025 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 19 12:26:46.914635 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.914617 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 19 12:26:46.914856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.914842 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 19 12:26:46.914971 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.914952 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 19 12:26:46.915046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.914952 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 19 12:26:46.915046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.914989 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4pd8c\"" Sep 19 12:26:46.915206 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.915192 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 19 12:26:46.921711 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.921666 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 19 12:26:46.923210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:46.923190 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c88d59dd5-tmxrc"] Sep 19 12:26:47.061503 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-oauth-config\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.061643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-console-config\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.061643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061569 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-service-ca\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.061643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061587 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-oauth-serving-cert\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.061643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdvz\" (UniqueName: \"kubernetes.io/projected/a35349a7-a8e7-4675-9453-6ef257968986-kube-api-access-rbdvz\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.061643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-trusted-ca-bundle\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.061940 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.061767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-serving-cert\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162719 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-oauth-config\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-console-config\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-service-ca\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-oauth-serving-cert\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdvz\" (UniqueName: \"kubernetes.io/projected/a35349a7-a8e7-4675-9453-6ef257968986-kube-api-access-rbdvz\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-trusted-ca-bundle\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.162990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.162920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-serving-cert\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.163634 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.163580 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-service-ca\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.163795 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.163771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-oauth-serving-cert\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.163869 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.163808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-console-config\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.163869 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.163808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-trusted-ca-bundle\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.165352 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.165332 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-serving-cert\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.165427 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.165386 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-oauth-config\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.170696 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.170660 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdvz\" (UniqueName: \"kubernetes.io/projected/a35349a7-a8e7-4675-9453-6ef257968986-kube-api-access-rbdvz\") pod \"console-7c88d59dd5-tmxrc\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.221475 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.221437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:47.499419 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:47.499393 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c88d59dd5-tmxrc"] Sep 19 12:26:47.501799 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:26:47.501769 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35349a7_a8e7_4675_9453_6ef257968986.slice/crio-6de96344b7e7a6ad4648f8b2146da541a1d311a3e1c327875e7e46c73ec5c1fb WatchSource:0}: Error finding container 6de96344b7e7a6ad4648f8b2146da541a1d311a3e1c327875e7e46c73ec5c1fb: Status 404 returned error can't find the container with id 6de96344b7e7a6ad4648f8b2146da541a1d311a3e1c327875e7e46c73ec5c1fb Sep 19 12:26:48.200610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:48.200571 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88d59dd5-tmxrc" event={"ID":"a35349a7-a8e7-4675-9453-6ef257968986","Type":"ContainerStarted","Data":"6de96344b7e7a6ad4648f8b2146da541a1d311a3e1c327875e7e46c73ec5c1fb"} Sep 19 12:26:52.222344 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:52.221829 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88d59dd5-tmxrc" event={"ID":"a35349a7-a8e7-4675-9453-6ef257968986","Type":"ContainerStarted","Data":"a63839677715075b51f91e36e10b4d98e028f96f8fbefdef19bdc26e2a3389b9"} Sep 19 12:26:52.244185 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:52.243197 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c88d59dd5-tmxrc" podStartSLOduration=1.6726874330000001 podStartE2EDuration="6.243176445s" totalImagesPullingTime="4.570489012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:26:52.242744871 +0000 UTC m=+107.849595477" watchObservedRunningTime="2026-09-19 12:26:52.243176445 +0000 UTC m=+107.850027025" Sep 19 12:26:57.221847 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:57.221811 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:57.222205 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:57.221880 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:57.226520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:57.226500 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:26:57.244331 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:26:57.244308 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:27:02.263084 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:02.263000 2585 generic.go:356] "Generic (PLEG): container finished" podID="072899a6-72d1-489a-b564-01e58cb221ce" containerID="29aa3759a69c52a1908c6ae72b28e3b0f94c963bed17b3278156ffe4c3d5336c" exitCode=0 Sep 19 12:27:02.263084 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:02.263061 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" event={"ID":"072899a6-72d1-489a-b564-01e58cb221ce","Type":"ContainerDied","Data":"29aa3759a69c52a1908c6ae72b28e3b0f94c963bed17b3278156ffe4c3d5336c"} Sep 19 12:27:02.263487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:02.263385 2585 scope.go:117] "RemoveContainer" containerID="29aa3759a69c52a1908c6ae72b28e3b0f94c963bed17b3278156ffe4c3d5336c" Sep 19 12:27:03.268799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:03.268764 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qrv2d" event={"ID":"072899a6-72d1-489a-b564-01e58cb221ce","Type":"ContainerStarted","Data":"8921574cbaaaa96daa0b2c1dd07a6bede331fecf4ef9f1c40e432c57450df1a7"} Sep 19 12:27:04.055627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.055587 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" podUID="bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" containerName="registry" containerID="cri-o://3fa17e048c6a3eff3044e9691c77a7bd3e6b9f6ff681322ff2b9f6144a97fa35" gracePeriod=30 Sep 19 12:27:04.274314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.274288 2585 generic.go:356] "Generic (PLEG): container finished" podID="bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" containerID="3fa17e048c6a3eff3044e9691c77a7bd3e6b9f6ff681322ff2b9f6144a97fa35" exitCode=0 Sep 19 12:27:04.274642 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.274357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" event={"ID":"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a","Type":"ContainerDied","Data":"3fa17e048c6a3eff3044e9691c77a7bd3e6b9f6ff681322ff2b9f6144a97fa35"} Sep 19 12:27:04.303502 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.303478 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:27:04.401935 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.401856 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-bound-sa-token\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.401935 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.401900 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-ca-trust-extracted\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.401936 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.401973 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-image-registry-private-configuration\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.401999 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv6q6\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-kube-api-access-zv6q6\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.402020 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-installation-pull-secrets\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.402069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-trusted-ca\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.402192 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-certificates\") pod \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\" (UID: \"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a\") " Sep 19 12:27:04.402889 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.402838 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:27:04.402889 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.402851 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:27:04.404966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.404910 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:04.404966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.404943 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:27:04.405180 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.405113 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:27:04.405180 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.405148 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:04.405263 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.405212 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-kube-api-access-zv6q6" (OuterVolumeSpecName: "kube-api-access-zv6q6") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "kube-api-access-zv6q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:27:04.410812 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.410787 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" (UID: "bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:27:04.503601 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503579 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-trusted-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503601 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503599 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-certificates\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503609 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-bound-sa-token\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503617 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-ca-trust-extracted\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503625 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-registry-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503634 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-image-registry-private-configuration\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503643 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6q6\" (UniqueName: \"kubernetes.io/projected/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-kube-api-access-zv6q6\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:04.503747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:04.503651 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a-installation-pull-secrets\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:05.279814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:05.279770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" event={"ID":"bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a","Type":"ContainerDied","Data":"726aa292af2c80f91b190a5f7df73143ad697d52a288ec67c92ce3523dd9b8fe"} Sep 19 12:27:05.280230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:05.279830 2585 scope.go:117] "RemoveContainer" containerID="3fa17e048c6a3eff3044e9691c77a7bd3e6b9f6ff681322ff2b9f6144a97fa35" Sep 19 12:27:05.280230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:05.279788 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56588f6dcc-zsg9v" Sep 19 12:27:05.297602 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:05.297579 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56588f6dcc-zsg9v"] Sep 19 12:27:05.302321 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:05.302303 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56588f6dcc-zsg9v"] Sep 19 12:27:06.578832 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:06.578299 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" path="/var/lib/kubelet/pods/bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a/volumes" Sep 19 12:27:11.305624 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:11.305591 2585 generic.go:356] "Generic (PLEG): container finished" podID="725756c5-21c2-4f17-b078-221ded0ffd0b" containerID="429b585aed59a83c4c51872c0831d9270eb3f5a202b37a6ac9832e155a040d61" exitCode=0 Sep 19 12:27:11.305979 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:11.305632 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" event={"ID":"725756c5-21c2-4f17-b078-221ded0ffd0b","Type":"ContainerDied","Data":"429b585aed59a83c4c51872c0831d9270eb3f5a202b37a6ac9832e155a040d61"} Sep 19 12:27:11.305979 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:11.305965 2585 scope.go:117] "RemoveContainer" containerID="429b585aed59a83c4c51872c0831d9270eb3f5a202b37a6ac9832e155a040d61" Sep 19 12:27:12.311502 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:12.311468 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-s66kr" event={"ID":"725756c5-21c2-4f17-b078-221ded0ffd0b","Type":"ContainerStarted","Data":"c23d147adc2a18a11ad68e578adbf7e2bb7624db0418ef896ac0ecf3c2f6eea3"} Sep 19 12:27:17.332806 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:17.332776 2585 generic.go:356] "Generic (PLEG): container finished" podID="5eec9ccb-6d80-4c68-82d5-1cd49377dfdf" containerID="e6e4f32f200d40a89493b6008236b0adb8dfaebafdb56a1a5aa27f9b0007afdc" exitCode=0 Sep 19 12:27:17.333204 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:17.332817 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" event={"ID":"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf","Type":"ContainerDied","Data":"e6e4f32f200d40a89493b6008236b0adb8dfaebafdb56a1a5aa27f9b0007afdc"} Sep 19 12:27:17.333204 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:17.333099 2585 scope.go:117] "RemoveContainer" containerID="e6e4f32f200d40a89493b6008236b0adb8dfaebafdb56a1a5aa27f9b0007afdc" Sep 19 12:27:18.339701 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:18.339645 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-n6jfv" event={"ID":"5eec9ccb-6d80-4c68-82d5-1cd49377dfdf","Type":"ContainerStarted","Data":"e775eb69d369f2962da977d0ae19044ca2ed421f1792a7f51db36ee40b7bcb41"} Sep 19 12:27:48.257438 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257360 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:27:48.257926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257623 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="alertmanager" containerID="cri-o://1bbc8c9abf6f492b73fdf9635954b02a605597f4387fd27e6b4c377e88e4b616" gracePeriod=120 Sep 19 12:27:48.257926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257726 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-web" containerID="cri-o://88554e4771896767f2ade46e430c59083bbd1a25902c0c336937cf0ba58a5cd6" gracePeriod=120 Sep 19 12:27:48.257926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257730 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-metric" containerID="cri-o://48c593c90c189f387e490999b5a6ad085fd9507be77accfa318a7f7dbcc26c57" gracePeriod=120 Sep 19 12:27:48.257926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257795 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="prom-label-proxy" containerID="cri-o://0152d3cf616a148bcbb04c41f2fcd0bbd5594916780f92130b19c2c66fea06b3" gracePeriod=120 Sep 19 12:27:48.257926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257798 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="config-reloader" containerID="cri-o://b89fae70fed4f964c67b346b322f78d7d65027017ff3483f8d98e2bc38727847" gracePeriod=120 Sep 19 12:27:48.257926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.257754 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy" containerID="cri-o://a9ec795db45d1f9b97e53b80c8ada9a0f8d72e5b979e57b78049ba6a2c736429" gracePeriod=120 Sep 19 12:27:48.477788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477755 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="0152d3cf616a148bcbb04c41f2fcd0bbd5594916780f92130b19c2c66fea06b3" exitCode=0 Sep 19 12:27:48.477788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477782 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="48c593c90c189f387e490999b5a6ad085fd9507be77accfa318a7f7dbcc26c57" exitCode=0 Sep 19 12:27:48.477788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477789 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="a9ec795db45d1f9b97e53b80c8ada9a0f8d72e5b979e57b78049ba6a2c736429" exitCode=0 Sep 19 12:27:48.477788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477795 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="b89fae70fed4f964c67b346b322f78d7d65027017ff3483f8d98e2bc38727847" exitCode=0 Sep 19 12:27:48.478032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477804 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="1bbc8c9abf6f492b73fdf9635954b02a605597f4387fd27e6b4c377e88e4b616" exitCode=0 Sep 19 12:27:48.478032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477826 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"0152d3cf616a148bcbb04c41f2fcd0bbd5594916780f92130b19c2c66fea06b3"} Sep 19 12:27:48.478032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"48c593c90c189f387e490999b5a6ad085fd9507be77accfa318a7f7dbcc26c57"} Sep 19 12:27:48.478032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477957 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"a9ec795db45d1f9b97e53b80c8ada9a0f8d72e5b979e57b78049ba6a2c736429"} Sep 19 12:27:48.478032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.477980 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"b89fae70fed4f964c67b346b322f78d7d65027017ff3483f8d98e2bc38727847"} Sep 19 12:27:48.478032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:48.478002 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"1bbc8c9abf6f492b73fdf9635954b02a605597f4387fd27e6b4c377e88e4b616"} Sep 19 12:27:49.487027 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.487000 2585 generic.go:356] "Generic (PLEG): container finished" podID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerID="88554e4771896767f2ade46e430c59083bbd1a25902c0c336937cf0ba58a5cd6" exitCode=0 Sep 19 12:27:49.487314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.487041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"88554e4771896767f2ade46e430c59083bbd1a25902c0c336937cf0ba58a5cd6"} Sep 19 12:27:49.503733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.503715 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:49.638515 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638434 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-web\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638515 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638475 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-web-config\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638515 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638520 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-tls-assets\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638538 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-volume\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638556 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638572 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v8pm\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-kube-api-access-9v8pm\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638602 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638622 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638646 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-cluster-tls-config\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638676 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-trusted-ca-bundle\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638727 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-metrics-client-ca\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638752 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-main-db\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.638803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.638789 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-out\") pod \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\" (UID: \"056eea10-a03c-4c26-9b8d-4ed8cfcb930e\") " Sep 19 12:27:49.639967 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.639938 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:27:49.640484 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.640461 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:27:49.640937 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.640913 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:27:49.641550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.641516 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.641853 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.641823 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-volume" (OuterVolumeSpecName: "config-volume") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.642034 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.642009 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:27:49.642254 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.642226 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.642404 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.642368 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-kube-api-access-9v8pm" (OuterVolumeSpecName: "kube-api-access-9v8pm") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "kube-api-access-9v8pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:27:49.643117 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.643094 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-out" (OuterVolumeSpecName: "config-out") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:27:49.644428 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.644397 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.644536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.644447 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.646013 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.645992 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.653574 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.653551 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-web-config" (OuterVolumeSpecName: "web-config") pod "056eea10-a03c-4c26-9b8d-4ed8cfcb930e" (UID: "056eea10-a03c-4c26-9b8d-4ed8cfcb930e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:27:49.740210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740186 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740208 2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-web-config\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740218 2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-tls-assets\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740225 2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740234 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740241 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v8pm\" (UniqueName: \"kubernetes.io/projected/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-kube-api-access-9v8pm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740251 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740261 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-secret-alertmanager-main-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740269 2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-cluster-tls-config\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740278 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740286 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-metrics-client-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740294 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-alertmanager-main-db\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:49.740322 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:49.740302 2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/056eea10-a03c-4c26-9b8d-4ed8cfcb930e-config-out\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:27:50.494052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.494024 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.494512 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.494024 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"056eea10-a03c-4c26-9b8d-4ed8cfcb930e","Type":"ContainerDied","Data":"4171873762d3aa7df1a99005ea697b0c5ff8d629a67dcf65dbba98c5b72472f3"} Sep 19 12:27:50.494512 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.494170 2585 scope.go:117] "RemoveContainer" containerID="0152d3cf616a148bcbb04c41f2fcd0bbd5594916780f92130b19c2c66fea06b3" Sep 19 12:27:50.502058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.502038 2585 scope.go:117] "RemoveContainer" containerID="48c593c90c189f387e490999b5a6ad085fd9507be77accfa318a7f7dbcc26c57" Sep 19 12:27:50.509097 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.509078 2585 scope.go:117] "RemoveContainer" containerID="a9ec795db45d1f9b97e53b80c8ada9a0f8d72e5b979e57b78049ba6a2c736429" Sep 19 12:27:50.515875 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.515856 2585 scope.go:117] "RemoveContainer" containerID="88554e4771896767f2ade46e430c59083bbd1a25902c0c336937cf0ba58a5cd6" Sep 19 12:27:50.518004 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.517958 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:27:50.521603 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.521584 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:27:50.523496 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.523478 2585 scope.go:117] "RemoveContainer" containerID="b89fae70fed4f964c67b346b322f78d7d65027017ff3483f8d98e2bc38727847" Sep 19 12:27:50.530386 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.530370 2585 scope.go:117] "RemoveContainer" containerID="1bbc8c9abf6f492b73fdf9635954b02a605597f4387fd27e6b4c377e88e4b616" Sep 19 12:27:50.538268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.538253 2585 scope.go:117] "RemoveContainer" containerID="48c7d80b5a6b52e6f7af1771f2acbbab5f1fb53f10f43080bd9e0c17e0cae25c" Sep 19 12:27:50.545219 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545193 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:27:50.545669 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545652 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="config-reloader" Sep 19 12:27:50.545733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545674 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="config-reloader" Sep 19 12:27:50.545733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545709 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="init-config-reloader" Sep 19 12:27:50.545733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545719 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="init-config-reloader" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545734 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-web" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545744 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-web" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545754 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545762 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545782 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="prom-label-proxy" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545791 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="prom-label-proxy" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545806 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="alertmanager" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545815 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="alertmanager" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545823 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-metric" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545828 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-metric" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545838 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" containerName="registry" Sep 19 12:27:50.545848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545842 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" containerName="registry" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545912 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="prom-label-proxy" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545922 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9bf34a-d0e5-4ad8-936f-f417a8a50c2a" containerName="registry" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545929 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-web" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545934 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy-metric" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545940 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="kube-rbac-proxy" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545948 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="alertmanager" Sep 19 12:27:50.546230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.545954 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" containerName="config-reloader" Sep 19 12:27:50.549742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.549725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.552072 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552053 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 19 12:27:50.552162 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552054 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 19 12:27:50.552217 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552188 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 19 12:27:50.552269 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552188 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 19 12:27:50.552327 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552310 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xcqw4\"" Sep 19 12:27:50.552390 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552374 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 19 12:27:50.552805 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552785 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 19 12:27:50.552892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552824 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 19 12:27:50.552892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.552844 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 19 12:27:50.559036 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.559018 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 19 12:27:50.561912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.561892 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:27:50.577113 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.577067 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056eea10-a03c-4c26-9b8d-4ed8cfcb930e" path="/var/lib/kubelet/pods/056eea10-a03c-4c26-9b8d-4ed8cfcb930e/volumes" Sep 19 12:27:50.645942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.645924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/482bdbeb-d56a-44be-90e3-8b36f21b246a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.645949 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/482bdbeb-d56a-44be-90e3-8b36f21b246a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.645969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.645999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/482bdbeb-d56a-44be-90e3-8b36f21b246a-config-out\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646018 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646032 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646048 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646110 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/482bdbeb-d56a-44be-90e3-8b36f21b246a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646138 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/482bdbeb-d56a-44be-90e3-8b36f21b246a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646163 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-config-volume\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646180 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xbx\" (UniqueName: \"kubernetes.io/projected/482bdbeb-d56a-44be-90e3-8b36f21b246a-kube-api-access-w6xbx\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646206 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.646255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.646231 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-web-config\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747431 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/482bdbeb-d56a-44be-90e3-8b36f21b246a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747516 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/482bdbeb-d56a-44be-90e3-8b36f21b246a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747516 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747452 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747622 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/482bdbeb-d56a-44be-90e3-8b36f21b246a-config-out\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747678 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747678 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747737 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/482bdbeb-d56a-44be-90e3-8b36f21b246a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/482bdbeb-d56a-44be-90e3-8b36f21b246a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747792 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/482bdbeb-d56a-44be-90e3-8b36f21b246a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.747814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-config-volume\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.748059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747830 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xbx\" (UniqueName: \"kubernetes.io/projected/482bdbeb-d56a-44be-90e3-8b36f21b246a-kube-api-access-w6xbx\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.748059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747884 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.748059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.747909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-web-config\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.748709 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.748669 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/482bdbeb-d56a-44be-90e3-8b36f21b246a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.750857 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.750494 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/482bdbeb-d56a-44be-90e3-8b36f21b246a-config-out\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.750857 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.750730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.751091 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.751063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.751197 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.751112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-web-config\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.751197 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.751136 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.751406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.751385 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/482bdbeb-d56a-44be-90e3-8b36f21b246a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.751472 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.751455 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/482bdbeb-d56a-44be-90e3-8b36f21b246a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.751601 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.751582 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.752037 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.752021 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-config-volume\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.752913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.752897 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/482bdbeb-d56a-44be-90e3-8b36f21b246a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.758877 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.758862 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xbx\" (UniqueName: \"kubernetes.io/projected/482bdbeb-d56a-44be-90e3-8b36f21b246a-kube-api-access-w6xbx\") pod \"alertmanager-main-0\" (UID: \"482bdbeb-d56a-44be-90e3-8b36f21b246a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:50.858767 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:50.858717 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 19 12:27:51.143275 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:51.143199 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 19 12:27:51.147118 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:27:51.147087 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482bdbeb_d56a_44be_90e3_8b36f21b246a.slice/crio-0c4156cb5251570e0eac8702859f49085d93351d71cc56bec98befd4620be931 WatchSource:0}: Error finding container 0c4156cb5251570e0eac8702859f49085d93351d71cc56bec98befd4620be931: Status 404 returned error can't find the container with id 0c4156cb5251570e0eac8702859f49085d93351d71cc56bec98befd4620be931 Sep 19 12:27:51.499574 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:51.499501 2585 generic.go:356] "Generic (PLEG): container finished" podID="482bdbeb-d56a-44be-90e3-8b36f21b246a" containerID="9a549b65066991c89103b7b9fe39ea852f59b4df9a8ba281d62433a1b12a72e0" exitCode=0 Sep 19 12:27:51.499969 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:51.499588 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerDied","Data":"9a549b65066991c89103b7b9fe39ea852f59b4df9a8ba281d62433a1b12a72e0"} Sep 19 12:27:51.499969 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:51.499677 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"0c4156cb5251570e0eac8702859f49085d93351d71cc56bec98befd4620be931"} Sep 19 12:27:52.276552 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.276518 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54fffcbc5-84dwj"] Sep 19 12:27:52.278966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.278942 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.281781 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.281761 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 19 12:27:52.281894 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.281763 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 19 12:27:52.281894 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.281767 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 19 12:27:52.282186 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.282161 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 19 12:27:52.282325 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.282161 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6qgtl\"" Sep 19 12:27:52.282570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.282548 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 19 12:27:52.287281 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.287260 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 19 12:27:52.302070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.302040 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54fffcbc5-84dwj"] Sep 19 12:27:52.360607 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68jr\" (UniqueName: \"kubernetes.io/projected/c92e16cb-8664-43f5-8339-476b93ff0a82-kube-api-access-q68jr\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360762 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360611 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-telemeter-client-tls\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360762 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-federate-client-tls\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360762 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360658 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360762 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-serving-certs-ca-bundle\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360935 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360776 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360935 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-secret-telemeter-client\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.360935 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.360884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-metrics-client-ca\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462113 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-secret-telemeter-client\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462224 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-metrics-client-ca\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462224 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462180 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68jr\" (UniqueName: \"kubernetes.io/projected/c92e16cb-8664-43f5-8339-476b93ff0a82-kube-api-access-q68jr\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462224 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-telemeter-client-tls\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462384 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-federate-client-tls\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462384 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462384 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-serving-certs-ca-bundle\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.462384 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.462344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.463151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.463047 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-serving-certs-ca-bundle\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.463151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.463176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-metrics-client-ca\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.463435 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.463347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92e16cb-8664-43f5-8339-476b93ff0a82-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.464792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.464767 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-secret-telemeter-client\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.464889 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.464837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.464953 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.464921 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-federate-client-tls\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.465006 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.464996 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c92e16cb-8664-43f5-8339-476b93ff0a82-telemeter-client-tls\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.471140 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.471119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68jr\" (UniqueName: \"kubernetes.io/projected/c92e16cb-8664-43f5-8339-476b93ff0a82-kube-api-access-q68jr\") pod \"telemeter-client-54fffcbc5-84dwj\" (UID: \"c92e16cb-8664-43f5-8339-476b93ff0a82\") " pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.507561 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.507539 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"7ba46dad98de82988dcd5d1f3983cfe10b1e7990181297ee0e7d5d939a90c381"} Sep 19 12:27:52.507858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.507597 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"310b35af49032d6506280f9e4a8a1b9835ce19da9a764388fe43b1aa5ae24e7a"} Sep 19 12:27:52.507858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.507623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"c21a016a1fd2c65b7103461f959a47a6cf55f0cab3340dd8659ababc9491e901"} Sep 19 12:27:52.507858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.507661 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"8416bb35c7d30cf1592f6021b598fb0088afcaded319a1b89d559d32448f132a"} Sep 19 12:27:52.507858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.507710 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"8765bb3640546e3232249d2c45e6d58f9937024d91dd730c37fbba5ce8a75dd4"} Sep 19 12:27:52.507858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.507746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"482bdbeb-d56a-44be-90e3-8b36f21b246a","Type":"ContainerStarted","Data":"b4b1a8aef57ad54f11d249195a66e08aaf258389132b9ccd548d18f7914169e2"} Sep 19 12:27:52.536340 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.536303 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.521070371 podStartE2EDuration="2.536287294s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.216923ms" isStatefulPod=false podCreationTimestamp="2026-09-19 12:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:27:52.535471417 +0000 UTC m=+168.142322062" watchObservedRunningTime="2026-09-19 12:27:52.536287294 +0000 UTC m=+168.143137872" Sep 19 12:27:52.589118 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.589088 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" Sep 19 12:27:52.866768 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:52.866673 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54fffcbc5-84dwj"] Sep 19 12:27:52.870552 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:27:52.870531 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92e16cb_8664_43f5_8339_476b93ff0a82.slice/crio-dd7d34070edf6fa4373efb6222af2fa3336b5a04b21627c376fefc3ad16472e1 WatchSource:0}: Error finding container dd7d34070edf6fa4373efb6222af2fa3336b5a04b21627c376fefc3ad16472e1: Status 404 returned error can't find the container with id dd7d34070edf6fa4373efb6222af2fa3336b5a04b21627c376fefc3ad16472e1 Sep 19 12:27:53.514285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:53.514252 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" event={"ID":"c92e16cb-8664-43f5-8339-476b93ff0a82","Type":"ContainerStarted","Data":"dd7d34070edf6fa4373efb6222af2fa3336b5a04b21627c376fefc3ad16472e1"} Sep 19 12:27:55.527006 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:55.526976 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" event={"ID":"c92e16cb-8664-43f5-8339-476b93ff0a82","Type":"ContainerStarted","Data":"ab828329fca43ea00d5d5dc6e5f011518ae473ec32a984416aac5c7cedc64106"} Sep 19 12:27:55.527459 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:55.527028 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" event={"ID":"c92e16cb-8664-43f5-8339-476b93ff0a82","Type":"ContainerStarted","Data":"6e2268cb8b64e2ce2ce24383ba19c7776429ae5e9252744f0d867f45f7ce1550"} Sep 19 12:27:55.527459 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:55.527048 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" event={"ID":"c92e16cb-8664-43f5-8339-476b93ff0a82","Type":"ContainerStarted","Data":"f972a30b1df2cc53b215e3b6d3addbffc3492f4c8080f020249ecff78f983891"} Sep 19 12:27:55.555374 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:27:55.555325 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54fffcbc5-84dwj" podStartSLOduration=1.6839415070000001 podStartE2EDuration="3.55531038s" totalImagesPullingTime="1.871368873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:27:55.554084829 +0000 UTC m=+171.160935430" watchObservedRunningTime="2026-09-19 12:27:55.55531038 +0000 UTC m=+171.162160959" Sep 19 12:28:06.495796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:06.495761 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c88d59dd5-tmxrc"] Sep 19 12:28:31.519301 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.519250 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c88d59dd5-tmxrc" podUID="a35349a7-a8e7-4675-9453-6ef257968986" containerName="console" containerID="cri-o://a63839677715075b51f91e36e10b4d98e028f96f8fbefdef19bdc26e2a3389b9" gracePeriod=15 Sep 19 12:28:31.676919 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.676885 2585 generic.go:356] "Generic (PLEG): container finished" podID="a35349a7-a8e7-4675-9453-6ef257968986" containerID="a63839677715075b51f91e36e10b4d98e028f96f8fbefdef19bdc26e2a3389b9" exitCode=0 Sep 19 12:28:31.677091 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.676966 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88d59dd5-tmxrc" event={"ID":"a35349a7-a8e7-4675-9453-6ef257968986","Type":"ContainerDied","Data":"a63839677715075b51f91e36e10b4d98e028f96f8fbefdef19bdc26e2a3389b9"} Sep 19 12:28:31.759838 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.759814 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:28:31.853541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853499 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-oauth-config\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.853662 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853553 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-oauth-serving-cert\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.853662 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853575 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-service-ca\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.853662 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853605 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-trusted-ca-bundle\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.853662 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853642 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbdvz\" (UniqueName: \"kubernetes.io/projected/a35349a7-a8e7-4675-9453-6ef257968986-kube-api-access-rbdvz\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.853892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853721 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-serving-cert\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.853892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.853755 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-console-config\") pod \"a35349a7-a8e7-4675-9453-6ef257968986\" (UID: \"a35349a7-a8e7-4675-9453-6ef257968986\") " Sep 19 12:28:31.854071 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.854020 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:28:31.854071 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.854028 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-service-ca" (OuterVolumeSpecName: "service-ca") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:28:31.854071 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.854059 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:28:31.854286 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.854267 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-console-config" (OuterVolumeSpecName: "console-config") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:28:31.855907 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.855879 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:28:31.855907 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.855902 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35349a7-a8e7-4675-9453-6ef257968986-kube-api-access-rbdvz" (OuterVolumeSpecName: "kube-api-access-rbdvz") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "kube-api-access-rbdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:28:31.856066 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.855919 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a35349a7-a8e7-4675-9453-6ef257968986" (UID: "a35349a7-a8e7-4675-9453-6ef257968986"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:28:31.954790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954762 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-serving-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:31.954790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954789 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-console-config\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:31.954909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954799 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a35349a7-a8e7-4675-9453-6ef257968986-console-oauth-config\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:31.954909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954807 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-oauth-serving-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:31.954909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954815 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:31.954909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954823 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35349a7-a8e7-4675-9453-6ef257968986-trusted-ca-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:31.954909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:31.954830 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbdvz\" (UniqueName: \"kubernetes.io/projected/a35349a7-a8e7-4675-9453-6ef257968986-kube-api-access-rbdvz\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:28:32.682216 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:32.682179 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88d59dd5-tmxrc" event={"ID":"a35349a7-a8e7-4675-9453-6ef257968986","Type":"ContainerDied","Data":"6de96344b7e7a6ad4648f8b2146da541a1d311a3e1c327875e7e46c73ec5c1fb"} Sep 19 12:28:32.682216 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:32.682203 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88d59dd5-tmxrc" Sep 19 12:28:32.682626 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:32.682244 2585 scope.go:117] "RemoveContainer" containerID="a63839677715075b51f91e36e10b4d98e028f96f8fbefdef19bdc26e2a3389b9" Sep 19 12:28:32.699640 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:32.699620 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c88d59dd5-tmxrc"] Sep 19 12:28:32.703878 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:32.703854 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c88d59dd5-tmxrc"] Sep 19 12:28:34.577629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:28:34.577599 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a35349a7-a8e7-4675-9453-6ef257968986" path="/var/lib/kubelet/pods/a35349a7-a8e7-4675-9453-6ef257968986/volumes" Sep 19 12:29:13.531501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.531464 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d4bc4fdff-drbsc"] Sep 19 12:29:13.532139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.531975 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35349a7-a8e7-4675-9453-6ef257968986" containerName="console" Sep 19 12:29:13.532139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.531995 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35349a7-a8e7-4675-9453-6ef257968986" containerName="console" Sep 19 12:29:13.532139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.532079 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35349a7-a8e7-4675-9453-6ef257968986" containerName="console" Sep 19 12:29:13.535043 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.535018 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.537741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.537717 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 19 12:29:13.537882 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.537846 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 19 12:29:13.537972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.537928 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 19 12:29:13.537972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.537963 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 19 12:29:13.538555 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.538531 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 19 12:29:13.539035 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.539016 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 19 12:29:13.539100 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.539055 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 19 12:29:13.539150 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.539016 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4pd8c\"" Sep 19 12:29:13.543314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.543033 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 19 12:29:13.545943 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.545923 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4bc4fdff-drbsc"] Sep 19 12:29:13.639250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639223 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-service-ca\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.639363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639291 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-config\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.639363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-serving-cert\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.639434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639387 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnl6f\" (UniqueName: \"kubernetes.io/projected/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-kube-api-access-gnl6f\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.639473 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-oauth-config\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.639511 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-oauth-serving-cert\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.639543 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.639511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-trusted-ca-bundle\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740471 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-oauth-serving-cert\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740567 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740481 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-trusted-ca-bundle\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740567 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-service-ca\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740650 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-config\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740707 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-serving-cert\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740762 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnl6f\" (UniqueName: \"kubernetes.io/projected/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-kube-api-access-gnl6f\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.740814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.740783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-oauth-config\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.741210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.741174 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-oauth-serving-cert\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.741511 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.741207 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-service-ca\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.741511 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.741245 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-config\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.741511 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.741309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-trusted-ca-bundle\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.743063 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.743046 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-serving-cert\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.743161 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.743143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-oauth-config\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.748046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.748020 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnl6f\" (UniqueName: \"kubernetes.io/projected/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-kube-api-access-gnl6f\") pod \"console-6d4bc4fdff-drbsc\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:13.847217 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:13.847194 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:14.118930 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:14.118908 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4bc4fdff-drbsc"] Sep 19 12:29:14.121358 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:29:14.121330 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6570bbc3_57b3_42cf_ace3_dc2c67ea4685.slice/crio-8268f3ca912763e5615abbb400fd3c3435d01c173d7a6d2fda3c50cc008a2d12 WatchSource:0}: Error finding container 8268f3ca912763e5615abbb400fd3c3435d01c173d7a6d2fda3c50cc008a2d12: Status 404 returned error can't find the container with id 8268f3ca912763e5615abbb400fd3c3435d01c173d7a6d2fda3c50cc008a2d12 Sep 19 12:29:14.850240 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:14.850206 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4bc4fdff-drbsc" event={"ID":"6570bbc3-57b3-42cf-ace3-dc2c67ea4685","Type":"ContainerStarted","Data":"72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c"} Sep 19 12:29:14.850600 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:14.850258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4bc4fdff-drbsc" event={"ID":"6570bbc3-57b3-42cf-ace3-dc2c67ea4685","Type":"ContainerStarted","Data":"8268f3ca912763e5615abbb400fd3c3435d01c173d7a6d2fda3c50cc008a2d12"} Sep 19 12:29:14.865872 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:14.865822 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d4bc4fdff-drbsc" podStartSLOduration=1.865804495 podStartE2EDuration="1.865804495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:29:14.865425085 +0000 UTC m=+250.472275676" watchObservedRunningTime="2026-09-19 12:29:14.865804495 +0000 UTC m=+250.472655074" Sep 19 12:29:23.847631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:23.847556 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:23.847631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:23.847619 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:23.852405 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:23.852382 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:29:23.890919 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:29:23.890898 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:30:06.443076 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:06.443050 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:30:06.450580 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:06.450548 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:30:06.453859 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:06.453829 2585 kubelet.go:1648] "Image garbage collection succeeded" Sep 19 12:30:16.244260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.244229 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8"] Sep 19 12:30:16.247446 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.247425 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.249868 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.249844 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:30:16.249868 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.249860 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 19 12:30:16.250026 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.249848 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m68gr\"" Sep 19 12:30:16.259607 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.259588 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8"] Sep 19 12:30:16.355095 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.355067 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbef8097-e176-40b9-b408-8338bbfcce10-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s7kv8\" (UID: \"cbef8097-e176-40b9-b408-8338bbfcce10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.355230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.355129 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7g4\" (UniqueName: \"kubernetes.io/projected/cbef8097-e176-40b9-b408-8338bbfcce10-kube-api-access-bl7g4\") pod \"cert-manager-operator-controller-manager-55c444bf86-s7kv8\" (UID: \"cbef8097-e176-40b9-b408-8338bbfcce10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.455724 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.455676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7g4\" (UniqueName: \"kubernetes.io/projected/cbef8097-e176-40b9-b408-8338bbfcce10-kube-api-access-bl7g4\") pod \"cert-manager-operator-controller-manager-55c444bf86-s7kv8\" (UID: \"cbef8097-e176-40b9-b408-8338bbfcce10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.455860 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.455741 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbef8097-e176-40b9-b408-8338bbfcce10-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s7kv8\" (UID: \"cbef8097-e176-40b9-b408-8338bbfcce10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.456069 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.456051 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbef8097-e176-40b9-b408-8338bbfcce10-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s7kv8\" (UID: \"cbef8097-e176-40b9-b408-8338bbfcce10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.467223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.467194 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7g4\" (UniqueName: \"kubernetes.io/projected/cbef8097-e176-40b9-b408-8338bbfcce10-kube-api-access-bl7g4\") pod \"cert-manager-operator-controller-manager-55c444bf86-s7kv8\" (UID: \"cbef8097-e176-40b9-b408-8338bbfcce10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.556808 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.556781 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" Sep 19 12:30:16.838704 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.838663 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8"] Sep 19 12:30:16.841316 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:30:16.841289 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbef8097_e176_40b9_b408_8338bbfcce10.slice/crio-391b2560bbd62b3f3c3fb56170784ca1b76148ec206b3bb3a429fc84bfbe43a9 WatchSource:0}: Error finding container 391b2560bbd62b3f3c3fb56170784ca1b76148ec206b3bb3a429fc84bfbe43a9: Status 404 returned error can't find the container with id 391b2560bbd62b3f3c3fb56170784ca1b76148ec206b3bb3a429fc84bfbe43a9 Sep 19 12:30:16.844640 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:16.844623 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 12:30:17.106785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:17.106707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" event={"ID":"cbef8097-e176-40b9-b408-8338bbfcce10","Type":"ContainerStarted","Data":"391b2560bbd62b3f3c3fb56170784ca1b76148ec206b3bb3a429fc84bfbe43a9"} Sep 19 12:30:20.122821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.122790 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" event={"ID":"cbef8097-e176-40b9-b408-8338bbfcce10","Type":"ContainerStarted","Data":"10ac67ed21905b7e4d767b4a1d82835d52904518c8e6b28cf4ca6706825d171a"} Sep 19 12:30:20.144218 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.144148 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s7kv8" podStartSLOduration=1.822317062 podStartE2EDuration="4.144128352s" totalImagesPullingTime="2.32181129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:30:20.142316382 +0000 UTC m=+315.749166984" watchObservedRunningTime="2026-09-19 12:30:20.144128352 +0000 UTC m=+315.750978932" Sep 19 12:30:20.684703 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.684656 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx"] Sep 19 12:30:20.688492 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.688460 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.690721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.690696 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2v6sm\"" Sep 19 12:30:20.690721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.690717 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 19 12:30:20.691456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.691423 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 19 12:30:20.701564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.701534 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx"] Sep 19 12:30:20.792390 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.792362 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.792538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.792405 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7h4\" (UniqueName: \"kubernetes.io/projected/f1e85672-c4f4-485c-afe9-90a7ea039573-kube-api-access-jz7h4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.792538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.792518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.893052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.893020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.893164 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.893063 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7h4\" (UniqueName: \"kubernetes.io/projected/f1e85672-c4f4-485c-afe9-90a7ea039573-kube-api-access-jz7h4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.893164 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.893140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.893391 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.893370 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.893481 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.893462 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.911993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.911971 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7h4\" (UniqueName: \"kubernetes.io/projected/f1e85672-c4f4-485c-afe9-90a7ea039573-kube-api-access-jz7h4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:20.998008 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:20.997956 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:21.269382 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:21.269309 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx"] Sep 19 12:30:21.272411 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:30:21.272382 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e85672_c4f4_485c_afe9_90a7ea039573.slice/crio-a41f8272cba6cfcc09c4c17a3dde460f5cbafb9ab5ec0463198fe534ea683d63 WatchSource:0}: Error finding container a41f8272cba6cfcc09c4c17a3dde460f5cbafb9ab5ec0463198fe534ea683d63: Status 404 returned error can't find the container with id a41f8272cba6cfcc09c4c17a3dde460f5cbafb9ab5ec0463198fe534ea683d63 Sep 19 12:30:22.133738 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.133705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" event={"ID":"f1e85672-c4f4-485c-afe9-90a7ea039573","Type":"ContainerStarted","Data":"a41f8272cba6cfcc09c4c17a3dde460f5cbafb9ab5ec0463198fe534ea683d63"} Sep 19 12:30:22.183346 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.183319 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9"] Sep 19 12:30:22.186819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.186797 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.189046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.189022 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 19 12:30:22.189136 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.189075 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 19 12:30:22.189863 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.189844 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7pshn\"" Sep 19 12:30:22.195896 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.195872 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9"] Sep 19 12:30:22.305133 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.305094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04faa0ec-9f8c-4c86-9564-9e404bae4ecf-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cqmx9\" (UID: \"04faa0ec-9f8c-4c86-9564-9e404bae4ecf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.305638 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.305255 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw58m\" (UniqueName: \"kubernetes.io/projected/04faa0ec-9f8c-4c86-9564-9e404bae4ecf-kube-api-access-zw58m\") pod \"cert-manager-cainjector-559cdd8c87-cqmx9\" (UID: \"04faa0ec-9f8c-4c86-9564-9e404bae4ecf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.406168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.406103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw58m\" (UniqueName: \"kubernetes.io/projected/04faa0ec-9f8c-4c86-9564-9e404bae4ecf-kube-api-access-zw58m\") pod \"cert-manager-cainjector-559cdd8c87-cqmx9\" (UID: \"04faa0ec-9f8c-4c86-9564-9e404bae4ecf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.406168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.406149 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04faa0ec-9f8c-4c86-9564-9e404bae4ecf-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cqmx9\" (UID: \"04faa0ec-9f8c-4c86-9564-9e404bae4ecf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.414294 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.414265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04faa0ec-9f8c-4c86-9564-9e404bae4ecf-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cqmx9\" (UID: \"04faa0ec-9f8c-4c86-9564-9e404bae4ecf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.414661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.414642 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw58m\" (UniqueName: \"kubernetes.io/projected/04faa0ec-9f8c-4c86-9564-9e404bae4ecf-kube-api-access-zw58m\") pod \"cert-manager-cainjector-559cdd8c87-cqmx9\" (UID: \"04faa0ec-9f8c-4c86-9564-9e404bae4ecf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.505596 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.505563 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" Sep 19 12:30:22.782345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:22.782271 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9"] Sep 19 12:30:22.785791 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:30:22.785765 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04faa0ec_9f8c_4c86_9564_9e404bae4ecf.slice/crio-d11b237d22952c26df456b09d7d8c6d76fd036614b53afce91860c1a7a056811 WatchSource:0}: Error finding container d11b237d22952c26df456b09d7d8c6d76fd036614b53afce91860c1a7a056811: Status 404 returned error can't find the container with id d11b237d22952c26df456b09d7d8c6d76fd036614b53afce91860c1a7a056811 Sep 19 12:30:23.139851 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:23.139818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" event={"ID":"04faa0ec-9f8c-4c86-9564-9e404bae4ecf","Type":"ContainerStarted","Data":"d11b237d22952c26df456b09d7d8c6d76fd036614b53afce91860c1a7a056811"} Sep 19 12:30:26.159058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:26.159017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" event={"ID":"04faa0ec-9f8c-4c86-9564-9e404bae4ecf","Type":"ContainerStarted","Data":"fbcc8e302a5c4e41d8c6ea405a4e7056993b3df75022678cf6cfaed42320e35c"} Sep 19 12:30:26.174800 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:26.174756 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cqmx9" podStartSLOduration=1.60705872 podStartE2EDuration="4.174736642s" totalImagesPullingTime="2.567677922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:30:26.173736089 +0000 UTC m=+321.780586696" watchObservedRunningTime="2026-09-19 12:30:26.174736642 +0000 UTC m=+321.781587221" Sep 19 12:30:28.170674 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:28.170643 2585 generic.go:356] "Generic (PLEG): container finished" podID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerID="295755bad621ef5c1e6521928c664ad8ad9cbd2c7baf8312c5563c7d79c7bdb4" exitCode=0 Sep 19 12:30:28.170978 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:28.170724 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" event={"ID":"f1e85672-c4f4-485c-afe9-90a7ea039573","Type":"ContainerDied","Data":"295755bad621ef5c1e6521928c664ad8ad9cbd2c7baf8312c5563c7d79c7bdb4"} Sep 19 12:30:30.183369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:30.183340 2585 generic.go:356] "Generic (PLEG): container finished" podID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerID="3add4c2be1c59d7a091c480f364ddb38689e5a494f3686a58a1fda56f865ae04" exitCode=0 Sep 19 12:30:30.183709 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:30.183415 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" event={"ID":"f1e85672-c4f4-485c-afe9-90a7ea039573","Type":"ContainerDied","Data":"3add4c2be1c59d7a091c480f364ddb38689e5a494f3686a58a1fda56f865ae04"} Sep 19 12:30:37.220301 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:37.220220 2585 generic.go:356] "Generic (PLEG): container finished" podID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerID="db7d8d400a19c598ffd4f9440d45a6a1c1227fc9457d8261eac2a571841cad43" exitCode=0 Sep 19 12:30:37.220605 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:37.220314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" event={"ID":"f1e85672-c4f4-485c-afe9-90a7ea039573","Type":"ContainerDied","Data":"db7d8d400a19c598ffd4f9440d45a6a1c1227fc9457d8261eac2a571841cad43"} Sep 19 12:30:38.353776 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.353752 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:38.446230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.446202 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-util\") pod \"f1e85672-c4f4-485c-afe9-90a7ea039573\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " Sep 19 12:30:38.446347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.446252 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz7h4\" (UniqueName: \"kubernetes.io/projected/f1e85672-c4f4-485c-afe9-90a7ea039573-kube-api-access-jz7h4\") pod \"f1e85672-c4f4-485c-afe9-90a7ea039573\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " Sep 19 12:30:38.446347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.446313 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-bundle\") pod \"f1e85672-c4f4-485c-afe9-90a7ea039573\" (UID: \"f1e85672-c4f4-485c-afe9-90a7ea039573\") " Sep 19 12:30:38.446736 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.446704 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-bundle" (OuterVolumeSpecName: "bundle") pod "f1e85672-c4f4-485c-afe9-90a7ea039573" (UID: "f1e85672-c4f4-485c-afe9-90a7ea039573"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:30:38.448553 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.448526 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e85672-c4f4-485c-afe9-90a7ea039573-kube-api-access-jz7h4" (OuterVolumeSpecName: "kube-api-access-jz7h4") pod "f1e85672-c4f4-485c-afe9-90a7ea039573" (UID: "f1e85672-c4f4-485c-afe9-90a7ea039573"). InnerVolumeSpecName "kube-api-access-jz7h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:30:38.451670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.451645 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-util" (OuterVolumeSpecName: "util") pod "f1e85672-c4f4-485c-afe9-90a7ea039573" (UID: "f1e85672-c4f4-485c-afe9-90a7ea039573"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:30:38.547406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.547383 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:30:38.547406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.547407 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e85672-c4f4-485c-afe9-90a7ea039573-util\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:30:38.547553 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:38.547416 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz7h4\" (UniqueName: \"kubernetes.io/projected/f1e85672-c4f4-485c-afe9-90a7ea039573-kube-api-access-jz7h4\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:30:39.231608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:39.231580 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" Sep 19 12:30:39.231788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:39.231585 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgmcjx" event={"ID":"f1e85672-c4f4-485c-afe9-90a7ea039573","Type":"ContainerDied","Data":"a41f8272cba6cfcc09c4c17a3dde460f5cbafb9ab5ec0463198fe534ea683d63"} Sep 19 12:30:39.231788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:39.231717 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41f8272cba6cfcc09c4c17a3dde460f5cbafb9ab5ec0463198fe534ea683d63" Sep 19 12:30:44.355878 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.355841 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2"] Sep 19 12:30:44.356234 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356209 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="extract" Sep 19 12:30:44.356234 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356222 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="extract" Sep 19 12:30:44.356303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356238 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="util" Sep 19 12:30:44.356303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356244 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="util" Sep 19 12:30:44.356303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356254 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="pull" Sep 19 12:30:44.356303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356259 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="pull" Sep 19 12:30:44.356422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.356314 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e85672-c4f4-485c-afe9-90a7ea039573" containerName="extract" Sep 19 12:30:44.411459 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.411426 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.411459 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.411439 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2"] Sep 19 12:30:44.413822 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.413799 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 19 12:30:44.413822 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.413814 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rkgqq\"" Sep 19 12:30:44.413972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.413831 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 19 12:30:44.487483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.487462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebb5ed03-5c19-404c-939e-0cd3732a1e55-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zjpj2\" (UID: \"ebb5ed03-5c19-404c-939e-0cd3732a1e55\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.487592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.487511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8v2\" (UniqueName: \"kubernetes.io/projected/ebb5ed03-5c19-404c-939e-0cd3732a1e55-kube-api-access-hd8v2\") pod \"openshift-lws-operator-fd8ccff4c-zjpj2\" (UID: \"ebb5ed03-5c19-404c-939e-0cd3732a1e55\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.588124 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.588100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8v2\" (UniqueName: \"kubernetes.io/projected/ebb5ed03-5c19-404c-939e-0cd3732a1e55-kube-api-access-hd8v2\") pod \"openshift-lws-operator-fd8ccff4c-zjpj2\" (UID: \"ebb5ed03-5c19-404c-939e-0cd3732a1e55\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.588245 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.588177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebb5ed03-5c19-404c-939e-0cd3732a1e55-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zjpj2\" (UID: \"ebb5ed03-5c19-404c-939e-0cd3732a1e55\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.588547 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.588529 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebb5ed03-5c19-404c-939e-0cd3732a1e55-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zjpj2\" (UID: \"ebb5ed03-5c19-404c-939e-0cd3732a1e55\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.595831 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.595814 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8v2\" (UniqueName: \"kubernetes.io/projected/ebb5ed03-5c19-404c-939e-0cd3732a1e55-kube-api-access-hd8v2\") pod \"openshift-lws-operator-fd8ccff4c-zjpj2\" (UID: \"ebb5ed03-5c19-404c-939e-0cd3732a1e55\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:44.720422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:44.720400 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" Sep 19 12:30:45.019046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:45.019023 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2"] Sep 19 12:30:45.021803 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:30:45.021771 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebb5ed03_5c19_404c_939e_0cd3732a1e55.slice/crio-5ef7f6837f82fa0d954d7b38b66fc4efa544c58b053d088967c37e6e79fe1a7a WatchSource:0}: Error finding container 5ef7f6837f82fa0d954d7b38b66fc4efa544c58b053d088967c37e6e79fe1a7a: Status 404 returned error can't find the container with id 5ef7f6837f82fa0d954d7b38b66fc4efa544c58b053d088967c37e6e79fe1a7a Sep 19 12:30:45.259480 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:45.259448 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" event={"ID":"ebb5ed03-5c19-404c-939e-0cd3732a1e55","Type":"ContainerStarted","Data":"5ef7f6837f82fa0d954d7b38b66fc4efa544c58b053d088967c37e6e79fe1a7a"} Sep 19 12:30:48.281652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:48.281620 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" event={"ID":"ebb5ed03-5c19-404c-939e-0cd3732a1e55","Type":"ContainerStarted","Data":"4da6afe2fd9c5b4ce86735c98bbd0575b9f8009a311797a710dbe21cc2db502c"} Sep 19 12:30:48.297988 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:30:48.297940 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zjpj2" podStartSLOduration=1.9484855140000001 podStartE2EDuration="4.297926591s" totalImagesPullingTime="2.349441077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:30:48.296070628 +0000 UTC m=+343.902921205" watchObservedRunningTime="2026-09-19 12:30:48.297926591 +0000 UTC m=+343.904777169" Sep 19 12:31:06.837853 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.837817 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj"] Sep 19 12:31:06.840770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.840743 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:06.844069 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.844044 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 19 12:31:06.844285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.844269 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 19 12:31:06.844595 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.844577 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 19 12:31:06.845199 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.845165 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 19 12:31:06.845711 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.845664 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bj9t6\"" Sep 19 12:31:06.855974 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.855950 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj"] Sep 19 12:31:06.967444 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.967412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:06.967576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.967458 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-webhook-cert\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:06.967576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.967533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-manifests\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:06.967576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.967557 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-tmp\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:06.967576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:06.967573 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfn5d\" (UniqueName: \"kubernetes.io/projected/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-kube-api-access-jfn5d\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.068515 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.068485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-manifests\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.068515 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.068514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-tmp\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.068813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.068531 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfn5d\" (UniqueName: \"kubernetes.io/projected/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-kube-api-access-jfn5d\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.068813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.068589 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.068813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.068614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-webhook-cert\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.068971 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.068896 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-manifests\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.071078 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.071046 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-tmp\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.071168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.071094 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-webhook-cert\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.071168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.071128 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.076433 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.076411 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfn5d\" (UniqueName: \"kubernetes.io/projected/1b9f2b63-5f0d-4034-8134-5e5e2edcc518-kube-api-access-jfn5d\") pod \"opendatahub-operator-controller-manager-774bc4964b-8mzpj\" (UID: \"1b9f2b63-5f0d-4034-8134-5e5e2edcc518\") " pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.156120 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.156067 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:07.436239 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:07.436216 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj"] Sep 19 12:31:07.438620 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:07.438594 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9f2b63_5f0d_4034_8134_5e5e2edcc518.slice/crio-ef946932f9a9d87ffb868212a14ca3700f20d12caf70432fa94a63d7579f5d7e WatchSource:0}: Error finding container ef946932f9a9d87ffb868212a14ca3700f20d12caf70432fa94a63d7579f5d7e: Status 404 returned error can't find the container with id ef946932f9a9d87ffb868212a14ca3700f20d12caf70432fa94a63d7579f5d7e Sep 19 12:31:08.382638 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:08.382591 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" event={"ID":"1b9f2b63-5f0d-4034-8134-5e5e2edcc518","Type":"ContainerStarted","Data":"ef946932f9a9d87ffb868212a14ca3700f20d12caf70432fa94a63d7579f5d7e"} Sep 19 12:31:11.400142 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:11.400104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" event={"ID":"1b9f2b63-5f0d-4034-8134-5e5e2edcc518","Type":"ContainerStarted","Data":"f98cdfd794df89f3a3ae6fe098a5ca8e1a9756024c141d737de1220f986aeee7"} Sep 19 12:31:12.406370 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:12.406334 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b9f2b63-5f0d-4034-8134-5e5e2edcc518" containerID="f98cdfd794df89f3a3ae6fe098a5ca8e1a9756024c141d737de1220f986aeee7" exitCode=0 Sep 19 12:31:12.406899 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:12.406423 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" event={"ID":"1b9f2b63-5f0d-4034-8134-5e5e2edcc518","Type":"ContainerDied","Data":"f98cdfd794df89f3a3ae6fe098a5ca8e1a9756024c141d737de1220f986aeee7"} Sep 19 12:31:13.413970 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:13.413935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" event={"ID":"1b9f2b63-5f0d-4034-8134-5e5e2edcc518","Type":"ContainerStarted","Data":"d502a1f77684b5ef5bf6336a893581d8d5b9a2c73d875a64d90fe908e5a23591"} Sep 19 12:31:13.414448 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:13.414067 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:13.448643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:13.448596 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" podStartSLOduration=3.00878737 podStartE2EDuration="7.448582466s" totalImagesPullingTime="3.371916049s" totalInitContainerRuntime="1.067879047s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:13.446416006 +0000 UTC m=+369.053266605" watchObservedRunningTime="2026-09-19 12:31:13.448582466 +0000 UTC m=+369.055433039" Sep 19 12:31:24.421481 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:24.421452 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-774bc4964b-8mzpj" Sep 19 12:31:26.440947 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.440918 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s"] Sep 19 12:31:26.444499 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.444483 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.447106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.447081 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9gb6c\"" Sep 19 12:31:26.447887 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.447868 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 19 12:31:26.447994 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.447935 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 19 12:31:26.455275 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.455250 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s"] Sep 19 12:31:26.527397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.527370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-tls-certs\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.527397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.527398 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjrj\" (UniqueName: \"kubernetes.io/projected/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-kube-api-access-dbjrj\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.527552 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.527450 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-tmp\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.628352 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.628325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-tmp\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.628469 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.628376 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-tls-certs\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.628523 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.628481 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjrj\" (UniqueName: \"kubernetes.io/projected/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-kube-api-access-dbjrj\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.630657 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.630638 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-tmp\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.630927 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.630910 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-tls-certs\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.635989 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.635971 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjrj\" (UniqueName: \"kubernetes.io/projected/6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a-kube-api-access-dbjrj\") pod \"kube-auth-proxy-5c9bc57f57-2m99s\" (UID: \"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a\") " pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:26.755802 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:26.755732 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" Sep 19 12:31:27.053748 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.053675 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s"] Sep 19 12:31:27.057763 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:27.057738 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6e34fa_2bd4_4fac_b7ba_3321b902cf5a.slice/crio-72478f0652d63127baf46c5a20cad3ca7d93f877948a6602b833b2b271f6fa96 WatchSource:0}: Error finding container 72478f0652d63127baf46c5a20cad3ca7d93f877948a6602b833b2b271f6fa96: Status 404 returned error can't find the container with id 72478f0652d63127baf46c5a20cad3ca7d93f877948a6602b833b2b271f6fa96 Sep 19 12:31:27.482790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.482706 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" event={"ID":"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a","Type":"ContainerStarted","Data":"72478f0652d63127baf46c5a20cad3ca7d93f877948a6602b833b2b271f6fa96"} Sep 19 12:31:27.707194 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.707164 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp"] Sep 19 12:31:27.711923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.711903 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.714290 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.714269 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 19 12:31:27.714402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.714294 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 19 12:31:27.714402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.714276 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 19 12:31:27.714596 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.714580 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 19 12:31:27.714672 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.714615 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9mz9k\"" Sep 19 12:31:27.727487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.727459 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp"] Sep 19 12:31:27.841194 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.841396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841241 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.841396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841324 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.841396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841351 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c45t\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-kube-api-access-4c45t\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.841660 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841458 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.841660 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841529 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb81f4ef-3244-469c-b57e-6246deb4574b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.841660 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.841554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942103 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942074 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb81f4ef-3244-469c-b57e-6246deb4574b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942103 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942133 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942153 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942191 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c45t\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-kube-api-access-4c45t\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942241 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.942756 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.942733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.945090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.945063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.945462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.945420 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.945462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.945453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb81f4ef-3244-469c-b57e-6246deb4574b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.945615 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.945552 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.950755 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.950666 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:27.951363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:27.951334 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c45t\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-kube-api-access-4c45t\") pod \"istiod-openshift-gateway-7b7995d5cd-2sbcp\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:28.022356 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:28.022321 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:28.313585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:28.313517 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp"] Sep 19 12:31:28.315290 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:28.315257 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb81f4ef_3244_469c_b57e_6246deb4574b.slice/crio-97bd7ce09569a81af972ad5d3f409ab7ce94017efbafa8b38337b1381334ef6f WatchSource:0}: Error finding container 97bd7ce09569a81af972ad5d3f409ab7ce94017efbafa8b38337b1381334ef6f: Status 404 returned error can't find the container with id 97bd7ce09569a81af972ad5d3f409ab7ce94017efbafa8b38337b1381334ef6f Sep 19 12:31:28.491225 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:28.491191 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" event={"ID":"fb81f4ef-3244-469c-b57e-6246deb4574b","Type":"ContainerStarted","Data":"97bd7ce09569a81af972ad5d3f409ab7ce94017efbafa8b38337b1381334ef6f"} Sep 19 12:31:30.504449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:30.504403 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" event={"ID":"6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a","Type":"ContainerStarted","Data":"e572e5ebef736adea209e4ede6b706ee156cee3d2c73117e5c8fae8c71eb1788"} Sep 19 12:31:30.525293 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:30.525248 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c9bc57f57-2m99s" podStartSLOduration=1.5673254939999999 podStartE2EDuration="4.52523263s" totalImagesPullingTime="2.957907136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:30.522247079 +0000 UTC m=+386.129097872" watchObservedRunningTime="2026-09-19 12:31:30.52523263 +0000 UTC m=+386.132083209" Sep 19 12:31:31.900627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:31.900586 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:31:31.900912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:31.900655 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:31:32.517451 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:32.517412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" event={"ID":"fb81f4ef-3244-469c-b57e-6246deb4574b","Type":"ContainerStarted","Data":"f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4"} Sep 19 12:31:32.517608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:32.517554 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:32.519489 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:32.519463 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2sbcp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 19 12:31:32.519603 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:32.519521 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 12:31:32.540058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:32.539999 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" podStartSLOduration=1.9574063430000002 podStartE2EDuration="5.539979434s" totalImagesPullingTime="3.582573091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:32.536439208 +0000 UTC m=+388.143289785" watchObservedRunningTime="2026-09-19 12:31:32.539979434 +0000 UTC m=+388.146830016" Sep 19 12:31:33.523547 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:33.523515 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2sbcp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 19 12:31:33.523987 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:33.523565 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 12:31:36.524262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:36.524226 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:31:36.937904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:36.937872 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4fpnf"] Sep 19 12:31:36.941409 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:36.941389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:36.944649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:36.944628 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 19 12:31:36.944649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:36.944633 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qlntd\"" Sep 19 12:31:36.958214 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:36.958183 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4fpnf"] Sep 19 12:31:37.021977 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.021946 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqtj\" (UniqueName: \"kubernetes.io/projected/7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98-kube-api-access-rgqtj\") pod \"odh-model-controller-7998c67dc4-4fpnf\" (UID: \"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98\") " pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.022128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.021987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98-cert\") pod \"odh-model-controller-7998c67dc4-4fpnf\" (UID: \"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98\") " pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.123338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.123302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqtj\" (UniqueName: \"kubernetes.io/projected/7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98-kube-api-access-rgqtj\") pod \"odh-model-controller-7998c67dc4-4fpnf\" (UID: \"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98\") " pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.123519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.123350 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98-cert\") pod \"odh-model-controller-7998c67dc4-4fpnf\" (UID: \"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98\") " pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.126098 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.126078 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98-cert\") pod \"odh-model-controller-7998c67dc4-4fpnf\" (UID: \"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98\") " pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.131928 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.131908 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqtj\" (UniqueName: \"kubernetes.io/projected/7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98-kube-api-access-rgqtj\") pod \"odh-model-controller-7998c67dc4-4fpnf\" (UID: \"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98\") " pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.251339 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.251260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:37.419750 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.419714 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pxnnj"] Sep 19 12:31:37.424252 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.424232 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.426899 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.426880 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8k2vs\"" Sep 19 12:31:37.426999 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.426936 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 19 12:31:37.434997 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.434970 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pxnnj"] Sep 19 12:31:37.523722 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.523678 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4fpnf"] Sep 19 12:31:37.525535 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:37.525509 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae87bc1_f5fa_4fd9_8bd7_f741f4450b98.slice/crio-e0251da67c26b4b87c2ec8fffe22b8c7d7d0edbf609ba73845dc194e387ea54c WatchSource:0}: Error finding container e0251da67c26b4b87c2ec8fffe22b8c7d7d0edbf609ba73845dc194e387ea54c: Status 404 returned error can't find the container with id e0251da67c26b4b87c2ec8fffe22b8c7d7d0edbf609ba73845dc194e387ea54c Sep 19 12:31:37.526655 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.526622 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870d3f82-f018-4b52-a1f1-4ac4b798469d-tls-certs\") pod \"model-serving-api-54d89bd5fc-pxnnj\" (UID: \"870d3f82-f018-4b52-a1f1-4ac4b798469d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.526787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.526747 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9zz\" (UniqueName: \"kubernetes.io/projected/870d3f82-f018-4b52-a1f1-4ac4b798469d-kube-api-access-7x9zz\") pod \"model-serving-api-54d89bd5fc-pxnnj\" (UID: \"870d3f82-f018-4b52-a1f1-4ac4b798469d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.542550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.542522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" event={"ID":"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98","Type":"ContainerStarted","Data":"e0251da67c26b4b87c2ec8fffe22b8c7d7d0edbf609ba73845dc194e387ea54c"} Sep 19 12:31:37.627478 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.627444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870d3f82-f018-4b52-a1f1-4ac4b798469d-tls-certs\") pod \"model-serving-api-54d89bd5fc-pxnnj\" (UID: \"870d3f82-f018-4b52-a1f1-4ac4b798469d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.627639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.627524 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9zz\" (UniqueName: \"kubernetes.io/projected/870d3f82-f018-4b52-a1f1-4ac4b798469d-kube-api-access-7x9zz\") pod \"model-serving-api-54d89bd5fc-pxnnj\" (UID: \"870d3f82-f018-4b52-a1f1-4ac4b798469d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.630178 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.630153 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870d3f82-f018-4b52-a1f1-4ac4b798469d-tls-certs\") pod \"model-serving-api-54d89bd5fc-pxnnj\" (UID: \"870d3f82-f018-4b52-a1f1-4ac4b798469d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.635589 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.635565 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9zz\" (UniqueName: \"kubernetes.io/projected/870d3f82-f018-4b52-a1f1-4ac4b798469d-kube-api-access-7x9zz\") pod \"model-serving-api-54d89bd5fc-pxnnj\" (UID: \"870d3f82-f018-4b52-a1f1-4ac4b798469d\") " pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:37.735676 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:37.735643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:38.008241 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:38.008212 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pxnnj"] Sep 19 12:31:38.010441 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:38.010414 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870d3f82_f018_4b52_a1f1_4ac4b798469d.slice/crio-9c99fd708ec3213e9739e8657619cf9a154ac542a81dbddb74b31a3e7e402756 WatchSource:0}: Error finding container 9c99fd708ec3213e9739e8657619cf9a154ac542a81dbddb74b31a3e7e402756: Status 404 returned error can't find the container with id 9c99fd708ec3213e9739e8657619cf9a154ac542a81dbddb74b31a3e7e402756 Sep 19 12:31:38.555125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:38.555067 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" event={"ID":"870d3f82-f018-4b52-a1f1-4ac4b798469d","Type":"ContainerStarted","Data":"9c99fd708ec3213e9739e8657619cf9a154ac542a81dbddb74b31a3e7e402756"} Sep 19 12:31:41.571549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:41.571509 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" event={"ID":"870d3f82-f018-4b52-a1f1-4ac4b798469d","Type":"ContainerStarted","Data":"fae6dcb58b59435f027d2de1bdc6f8730320462b827805170f9de7a584924f68"} Sep 19 12:31:41.572036 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:41.571578 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:41.573004 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:41.572982 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" event={"ID":"7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98","Type":"ContainerStarted","Data":"645553f8a808ebd7ad86cba66c0f032889db7af46457898f4086dc0c773d7d69"} Sep 19 12:31:41.573137 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:41.573113 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:41.593141 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:41.593103 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" podStartSLOduration=1.826108405 podStartE2EDuration="4.593089763s" totalImagesPullingTime="2.766981358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:41.591458014 +0000 UTC m=+397.198308622" watchObservedRunningTime="2026-09-19 12:31:41.593089763 +0000 UTC m=+397.199940342" Sep 19 12:31:41.609406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:41.609366 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" podStartSLOduration=2.354847139 podStartE2EDuration="5.609346485s" totalImagesPullingTime="3.254499346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:41.608121143 +0000 UTC m=+397.214971748" watchObservedRunningTime="2026-09-19 12:31:41.609346485 +0000 UTC m=+397.216197064" Sep 19 12:31:45.428248 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.428214 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7994d448f7-qb288"] Sep 19 12:31:45.438731 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.438705 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.446110 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.445662 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7994d448f7-qb288"] Sep 19 12:31:45.501475 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1065c848-981f-4604-a594-4fe04fd969d6-console-serving-cert\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.501652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501498 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-console-config\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.501652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501545 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-trusted-ca-bundle\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.501652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501603 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1065c848-981f-4604-a594-4fe04fd969d6-console-oauth-config\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.501652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501634 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-oauth-serving-cert\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.501932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501658 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkccv\" (UniqueName: \"kubernetes.io/projected/1065c848-981f-4604-a594-4fe04fd969d6-kube-api-access-nkccv\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.501932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.501743 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-service-ca\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602440 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602407 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-oauth-serving-cert\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602440 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkccv\" (UniqueName: \"kubernetes.io/projected/1065c848-981f-4604-a594-4fe04fd969d6-kube-api-access-nkccv\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602470 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-service-ca\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1065c848-981f-4604-a594-4fe04fd969d6-console-serving-cert\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-console-config\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602589 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-trusted-ca-bundle\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.602665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.602630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1065c848-981f-4604-a594-4fe04fd969d6-console-oauth-config\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.603256 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.603232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-oauth-serving-cert\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.603362 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.603267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-service-ca\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.603427 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.603367 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-console-config\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.603491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.603470 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1065c848-981f-4604-a594-4fe04fd969d6-trusted-ca-bundle\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.605070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.605049 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1065c848-981f-4604-a594-4fe04fd969d6-console-serving-cert\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.605207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.605187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1065c848-981f-4604-a594-4fe04fd969d6-console-oauth-config\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.619332 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.619314 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkccv\" (UniqueName: \"kubernetes.io/projected/1065c848-981f-4604-a594-4fe04fd969d6-kube-api-access-nkccv\") pod \"console-7994d448f7-qb288\" (UID: \"1065c848-981f-4604-a594-4fe04fd969d6\") " pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:45.748375 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:45.748318 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:46.037640 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:46.037616 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7994d448f7-qb288"] Sep 19 12:31:46.039646 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:46.039624 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1065c848_981f_4604_a594_4fe04fd969d6.slice/crio-d31698f41ffb8657f74407a91b78d5c2c22258a8b24e253bac2854a79eb25cd3 WatchSource:0}: Error finding container d31698f41ffb8657f74407a91b78d5c2c22258a8b24e253bac2854a79eb25cd3: Status 404 returned error can't find the container with id d31698f41ffb8657f74407a91b78d5c2c22258a8b24e253bac2854a79eb25cd3 Sep 19 12:31:46.598466 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:46.598435 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7994d448f7-qb288" event={"ID":"1065c848-981f-4604-a594-4fe04fd969d6","Type":"ContainerStarted","Data":"8954092aaec891079733cfe7d502cdf4edf1131e32df0529a3e1ac2f235b641f"} Sep 19 12:31:46.599012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:46.598484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7994d448f7-qb288" event={"ID":"1065c848-981f-4604-a594-4fe04fd969d6","Type":"ContainerStarted","Data":"d31698f41ffb8657f74407a91b78d5c2c22258a8b24e253bac2854a79eb25cd3"} Sep 19 12:31:46.646563 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:46.646496 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7994d448f7-qb288" podStartSLOduration=1.64647451 podStartE2EDuration="1.64647451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:46.643117805 +0000 UTC m=+402.249968399" watchObservedRunningTime="2026-09-19 12:31:46.64647451 +0000 UTC m=+402.253325084" Sep 19 12:31:47.353692 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.353661 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5"] Sep 19 12:31:47.357189 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.357174 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.360797 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.360777 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bwhc5\"" Sep 19 12:31:47.361083 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.361069 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 19 12:31:47.397545 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.397516 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr"] Sep 19 12:31:47.400729 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.400709 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5"] Sep 19 12:31:47.400840 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.400821 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:47.403197 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.403177 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 19 12:31:47.403292 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.403216 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kzcsk\"" Sep 19 12:31:47.413821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.413798 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr"] Sep 19 12:31:47.419823 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.419804 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc4281c1-7e85-41d6-8da0-45f3889bf2d8-cert\") pod \"kserve-controller-manager-d6f4b65b8-r8qf5\" (UID: \"cc4281c1-7e85-41d6-8da0-45f3889bf2d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.419918 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.419862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvht\" (UniqueName: \"kubernetes.io/projected/cc4281c1-7e85-41d6-8da0-45f3889bf2d8-kube-api-access-jwvht\") pod \"kserve-controller-manager-d6f4b65b8-r8qf5\" (UID: \"cc4281c1-7e85-41d6-8da0-45f3889bf2d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.521162 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.521132 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvht\" (UniqueName: \"kubernetes.io/projected/cc4281c1-7e85-41d6-8da0-45f3889bf2d8-kube-api-access-jwvht\") pod \"kserve-controller-manager-d6f4b65b8-r8qf5\" (UID: \"cc4281c1-7e85-41d6-8da0-45f3889bf2d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.521340 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.521189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49eba3e5-c592-4646-9ad9-4b444d8de961-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:47.521340 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.521262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc4281c1-7e85-41d6-8da0-45f3889bf2d8-cert\") pod \"kserve-controller-manager-d6f4b65b8-r8qf5\" (UID: \"cc4281c1-7e85-41d6-8da0-45f3889bf2d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.521340 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.521322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8jf\" (UniqueName: \"kubernetes.io/projected/49eba3e5-c592-4646-9ad9-4b444d8de961-kube-api-access-cv8jf\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:47.523862 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.523839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc4281c1-7e85-41d6-8da0-45f3889bf2d8-cert\") pod \"kserve-controller-manager-d6f4b65b8-r8qf5\" (UID: \"cc4281c1-7e85-41d6-8da0-45f3889bf2d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.576884 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.576856 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvht\" (UniqueName: \"kubernetes.io/projected/cc4281c1-7e85-41d6-8da0-45f3889bf2d8-kube-api-access-jwvht\") pod \"kserve-controller-manager-d6f4b65b8-r8qf5\" (UID: \"cc4281c1-7e85-41d6-8da0-45f3889bf2d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.622739 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.622654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49eba3e5-c592-4646-9ad9-4b444d8de961-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:47.623082 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:31:47.622783 2585 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 19 12:31:47.623082 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:31:47.622836 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49eba3e5-c592-4646-9ad9-4b444d8de961-cert podName:49eba3e5-c592-4646-9ad9-4b444d8de961 nodeName:}" failed. No retries permitted until 2026-09-19 12:31:48.12282242 +0000 UTC m=+403.729672977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/49eba3e5-c592-4646-9ad9-4b444d8de961-cert") pod "llmisvc-controller-manager-777c6bd7d5-lfshr" (UID: "49eba3e5-c592-4646-9ad9-4b444d8de961") : secret "llmisvc-webhook-server-cert" not found Sep 19 12:31:47.623082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.622782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8jf\" (UniqueName: \"kubernetes.io/projected/49eba3e5-c592-4646-9ad9-4b444d8de961-kube-api-access-cv8jf\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:47.637157 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.637135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8jf\" (UniqueName: \"kubernetes.io/projected/49eba3e5-c592-4646-9ad9-4b444d8de961-kube-api-access-cv8jf\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:47.667529 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.667505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:47.957114 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:47.957056 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5"] Sep 19 12:31:47.959676 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:47.959651 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4281c1_7e85_41d6_8da0_45f3889bf2d8.slice/crio-0f41697f924f0b5156331c2e735f1dbd0c2aa30f6327754d8233dd9a7d4b275b WatchSource:0}: Error finding container 0f41697f924f0b5156331c2e735f1dbd0c2aa30f6327754d8233dd9a7d4b275b: Status 404 returned error can't find the container with id 0f41697f924f0b5156331c2e735f1dbd0c2aa30f6327754d8233dd9a7d4b275b Sep 19 12:31:48.128738 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:48.128701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49eba3e5-c592-4646-9ad9-4b444d8de961-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:48.131299 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:48.131280 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49eba3e5-c592-4646-9ad9-4b444d8de961-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lfshr\" (UID: \"49eba3e5-c592-4646-9ad9-4b444d8de961\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:48.311078 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:48.311051 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:48.590637 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:48.590605 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr"] Sep 19 12:31:48.592306 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:31:48.592279 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49eba3e5_c592_4646_9ad9_4b444d8de961.slice/crio-ae57d4e0afe149de5bcb9892fed8a2fa9d81bd2c6551081620166bbac401e573 WatchSource:0}: Error finding container ae57d4e0afe149de5bcb9892fed8a2fa9d81bd2c6551081620166bbac401e573: Status 404 returned error can't find the container with id ae57d4e0afe149de5bcb9892fed8a2fa9d81bd2c6551081620166bbac401e573 Sep 19 12:31:48.610051 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:48.610025 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" event={"ID":"49eba3e5-c592-4646-9ad9-4b444d8de961","Type":"ContainerStarted","Data":"ae57d4e0afe149de5bcb9892fed8a2fa9d81bd2c6551081620166bbac401e573"} Sep 19 12:31:48.611131 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:48.611113 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" event={"ID":"cc4281c1-7e85-41d6-8da0-45f3889bf2d8","Type":"ContainerStarted","Data":"0f41697f924f0b5156331c2e735f1dbd0c2aa30f6327754d8233dd9a7d4b275b"} Sep 19 12:31:51.628886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:51.628801 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" event={"ID":"cc4281c1-7e85-41d6-8da0-45f3889bf2d8","Type":"ContainerStarted","Data":"bcb470a41728b443d3bc672ac7bc2883695a43d83ab3deb9f6d227b680e7a178"} Sep 19 12:31:51.629309 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:51.628946 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:31:51.630493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:51.630474 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" event={"ID":"49eba3e5-c592-4646-9ad9-4b444d8de961","Type":"ContainerStarted","Data":"af097d63b0ccd08c7244801fd3c22d1590a22ee6916515c526204c6b45276891"} Sep 19 12:31:51.630619 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:51.630604 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:31:51.671366 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:51.671322 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" podStartSLOduration=2.089451921 podStartE2EDuration="4.671309329s" totalImagesPullingTime="2.581857408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:51.669058053 +0000 UTC m=+407.275908658" watchObservedRunningTime="2026-09-19 12:31:51.671309329 +0000 UTC m=+407.278159929" Sep 19 12:31:51.710378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:51.710337 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" podStartSLOduration=2.007509459 podStartE2EDuration="4.71032374s" totalImagesPullingTime="2.702814281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:31:51.707541381 +0000 UTC m=+407.314391980" watchObservedRunningTime="2026-09-19 12:31:51.71032374 +0000 UTC m=+407.317174372" Sep 19 12:31:52.579975 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:52.579944 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4fpnf" Sep 19 12:31:52.581959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:52.581937 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pxnnj" Sep 19 12:31:55.748871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:55.748838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:55.749265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:55.748891 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:55.753364 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:55.753335 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:56.659492 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:56.659461 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7994d448f7-qb288" Sep 19 12:31:56.713260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:31:56.713221 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d4bc4fdff-drbsc"] Sep 19 12:32:01.419125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.419092 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp"] Sep 19 12:32:01.419581 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.419264 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerName="discovery" containerID="cri-o://f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4" gracePeriod=30 Sep 19 12:32:01.658278 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.658247 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:32:01.684646 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.684564 2585 generic.go:356] "Generic (PLEG): container finished" podID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerID="f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4" exitCode=0 Sep 19 12:32:01.684810 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.684658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" event={"ID":"fb81f4ef-3244-469c-b57e-6246deb4574b","Type":"ContainerDied","Data":"f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4"} Sep 19 12:32:01.684810 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.684665 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" Sep 19 12:32:01.684810 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.684728 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp" event={"ID":"fb81f4ef-3244-469c-b57e-6246deb4574b","Type":"ContainerDied","Data":"97bd7ce09569a81af972ad5d3f409ab7ce94017efbafa8b38337b1381334ef6f"} Sep 19 12:32:01.684810 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.684757 2585 scope.go:117] "RemoveContainer" containerID="f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4" Sep 19 12:32:01.694012 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.693985 2585 scope.go:117] "RemoveContainer" containerID="f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4" Sep 19 12:32:01.694332 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:32:01.694308 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4\": container with ID starting with f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4 not found: ID does not exist" containerID="f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4" Sep 19 12:32:01.694407 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.694341 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4"} err="failed to get container status \"f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4\": rpc error: code = NotFound desc = could not find container \"f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4\": container with ID starting with f0fe916344bf498dc87787e082c54d55558f32b77e87738dced8ce38902e00c4 not found: ID does not exist" Sep 19 12:32:01.745059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745029 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-dns-cert\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745226 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745094 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c45t\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-kube-api-access-4c45t\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745226 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745128 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-ca-configmap\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745226 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745148 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-cacerts\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745226 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745164 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb81f4ef-3244-469c-b57e-6246deb4574b-local-certs\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745452 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745303 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-token\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745452 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745377 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-kubeconfig\") pod \"fb81f4ef-3244-469c-b57e-6246deb4574b\" (UID: \"fb81f4ef-3244-469c-b57e-6246deb4574b\") " Sep 19 12:32:01.745608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745581 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:32:01.745954 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.745785 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-ca-configmap\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:01.747617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.747579 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-kube-api-access-4c45t" (OuterVolumeSpecName: "kube-api-access-4c45t") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "kube-api-access-4c45t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:01.747742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.747653 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-cacerts" (OuterVolumeSpecName: "cacerts") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:32:01.747970 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.747943 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb81f4ef-3244-469c-b57e-6246deb4574b-local-certs" (OuterVolumeSpecName: "local-certs") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:01.748075 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.747946 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:01.748135 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.748076 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:32:01.748193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.748173 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fb81f4ef-3244-469c-b57e-6246deb4574b" (UID: "fb81f4ef-3244-469c-b57e-6246deb4574b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:32:01.847250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.847216 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-csr-dns-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:01.847250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.847247 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c45t\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-kube-api-access-4c45t\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:01.847250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.847257 2585 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-cacerts\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:01.847410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.847265 2585 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb81f4ef-3244-469c-b57e-6246deb4574b-local-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:01.847410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.847274 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-token\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:01.847410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:01.847281 2585 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb81f4ef-3244-469c-b57e-6246deb4574b-istio-kubeconfig\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:02.007582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:02.007557 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp"] Sep 19 12:32:02.009382 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:02.009364 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2sbcp"] Sep 19 12:32:02.578463 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:02.578429 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" path="/var/lib/kubelet/pods/fb81f4ef-3244-469c-b57e-6246deb4574b/volumes" Sep 19 12:32:21.737669 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:21.737576 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d4bc4fdff-drbsc" podUID="6570bbc3-57b3-42cf-ace3-dc2c67ea4685" containerName="console" containerID="cri-o://72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c" gracePeriod=15 Sep 19 12:32:22.003405 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.003345 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:32:22.104041 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104017 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-oauth-serving-cert\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104041 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104051 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-config\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnl6f\" (UniqueName: \"kubernetes.io/projected/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-kube-api-access-gnl6f\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104115 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-serving-cert\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-trusted-ca-bundle\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104194 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-service-ca\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104218 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-oauth-config\") pod \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\" (UID: \"6570bbc3-57b3-42cf-ace3-dc2c67ea4685\") " Sep 19 12:32:22.104476 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104397 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:32:22.104536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104507 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-config" (OuterVolumeSpecName: "console-config") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:32:22.104610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104593 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-oauth-serving-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.104665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104616 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-config\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.104747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104713 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:32:22.104806 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.104781 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-service-ca" (OuterVolumeSpecName: "service-ca") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:32:22.106564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.106524 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:32:22.106658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.106593 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:32:22.106658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.106608 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-kube-api-access-gnl6f" (OuterVolumeSpecName: "kube-api-access-gnl6f") pod "6570bbc3-57b3-42cf-ace3-dc2c67ea4685" (UID: "6570bbc3-57b3-42cf-ace3-dc2c67ea4685"). InnerVolumeSpecName "kube-api-access-gnl6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:22.205892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.205868 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-trusted-ca-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.205892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.205889 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.206013 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.205899 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-oauth-config\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.206013 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.205909 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnl6f\" (UniqueName: \"kubernetes.io/projected/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-kube-api-access-gnl6f\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.206013 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.205917 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6570bbc3-57b3-42cf-ace3-dc2c67ea4685-console-serving-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:22.638992 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.638964 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lfshr" Sep 19 12:32:22.641910 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.641887 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r8qf5" Sep 19 12:32:22.789456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.789427 2585 generic.go:356] "Generic (PLEG): container finished" podID="6570bbc3-57b3-42cf-ace3-dc2c67ea4685" containerID="72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c" exitCode=0 Sep 19 12:32:22.789842 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.789471 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4bc4fdff-drbsc" Sep 19 12:32:22.789842 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.789528 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4bc4fdff-drbsc" event={"ID":"6570bbc3-57b3-42cf-ace3-dc2c67ea4685","Type":"ContainerDied","Data":"72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c"} Sep 19 12:32:22.789842 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.789574 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4bc4fdff-drbsc" event={"ID":"6570bbc3-57b3-42cf-ace3-dc2c67ea4685","Type":"ContainerDied","Data":"8268f3ca912763e5615abbb400fd3c3435d01c173d7a6d2fda3c50cc008a2d12"} Sep 19 12:32:22.789842 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.789594 2585 scope.go:117] "RemoveContainer" containerID="72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c" Sep 19 12:32:22.798825 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.798809 2585 scope.go:117] "RemoveContainer" containerID="72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c" Sep 19 12:32:22.799098 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:32:22.799077 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c\": container with ID starting with 72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c not found: ID does not exist" containerID="72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c" Sep 19 12:32:22.799165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.799110 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c"} err="failed to get container status \"72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c\": rpc error: code = NotFound desc = could not find container \"72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c\": container with ID starting with 72b255ca51ff247742a40834eef053ba882d4e111f4c77dc4b8dadd9fe59229c not found: ID does not exist" Sep 19 12:32:22.805695 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.805666 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d4bc4fdff-drbsc"] Sep 19 12:32:22.809081 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:22.809061 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d4bc4fdff-drbsc"] Sep 19 12:32:24.579030 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:24.578996 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6570bbc3-57b3-42cf-ace3-dc2c67ea4685" path="/var/lib/kubelet/pods/6570bbc3-57b3-42cf-ace3-dc2c67ea4685/volumes" Sep 19 12:32:42.003851 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.003818 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6"] Sep 19 12:32:42.004343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.004231 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6570bbc3-57b3-42cf-ace3-dc2c67ea4685" containerName="console" Sep 19 12:32:42.004343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.004242 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6570bbc3-57b3-42cf-ace3-dc2c67ea4685" containerName="console" Sep 19 12:32:42.004343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.004253 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerName="discovery" Sep 19 12:32:42.004343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.004258 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerName="discovery" Sep 19 12:32:42.004343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.004317 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6570bbc3-57b3-42cf-ace3-dc2c67ea4685" containerName="console" Sep 19 12:32:42.004343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.004326 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb81f4ef-3244-469c-b57e-6246deb4574b" containerName="discovery" Sep 19 12:32:42.007693 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.007665 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.009974 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.009933 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 19 12:32:42.010714 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.010674 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2v6sm\"" Sep 19 12:32:42.010714 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.010677 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 19 12:32:42.014244 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.014219 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6"] Sep 19 12:32:42.073919 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.073897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.074016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.073926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.074016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.073999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmzx\" (UniqueName: \"kubernetes.io/projected/68955ce7-a707-49b0-8162-3ac3057c33da-kube-api-access-pzmzx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.099595 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.099572 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts"] Sep 19 12:32:42.102320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.102304 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.110796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.110775 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts"] Sep 19 12:32:42.175262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175235 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.175363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.175363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175291 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.175363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175315 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.175471 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175386 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmzx\" (UniqueName: \"kubernetes.io/projected/68955ce7-a707-49b0-8162-3ac3057c33da-kube-api-access-pzmzx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.175471 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sm6\" (UniqueName: \"kubernetes.io/projected/1af75591-7757-494b-8957-e1fa3ddd0a4b-kube-api-access-f8sm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.175604 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175589 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.175643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.175625 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.186568 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.186546 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmzx\" (UniqueName: \"kubernetes.io/projected/68955ce7-a707-49b0-8162-3ac3057c33da-kube-api-access-pzmzx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.204547 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.204523 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p"] Sep 19 12:32:42.207361 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.207346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.215964 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.215939 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p"] Sep 19 12:32:42.276437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sm6\" (UniqueName: \"kubernetes.io/projected/1af75591-7757-494b-8957-e1fa3ddd0a4b-kube-api-access-f8sm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.276437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.276437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.276623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.276623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.276623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwgc\" (UniqueName: \"kubernetes.io/projected/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-kube-api-access-rpwgc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.276909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276890 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.276979 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.276935 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.286631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.286614 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sm6\" (UniqueName: \"kubernetes.io/projected/1af75591-7757-494b-8957-e1fa3ddd0a4b-kube-api-access-f8sm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.302907 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.302879 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr"] Sep 19 12:32:42.305953 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.305937 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.314196 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.314176 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr"] Sep 19 12:32:42.318781 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.318761 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:42.377236 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.377366 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.377366 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377283 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bl6\" (UniqueName: \"kubernetes.io/projected/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-kube-api-access-g8bl6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.377493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.377493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwgc\" (UniqueName: \"kubernetes.io/projected/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-kube-api-access-rpwgc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.377493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377440 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.377653 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377624 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.377748 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.377706 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.385984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.385962 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwgc\" (UniqueName: \"kubernetes.io/projected/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-kube-api-access-rpwgc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.411468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.411444 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:42.478261 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.478233 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bl6\" (UniqueName: \"kubernetes.io/projected/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-kube-api-access-g8bl6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.478367 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.478297 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.478367 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.478324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.478764 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.478744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.478815 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.478754 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.488274 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.488250 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bl6\" (UniqueName: \"kubernetes.io/projected/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-kube-api-access-g8bl6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.517546 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.517523 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:42.615665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.615638 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6"] Sep 19 12:32:42.616169 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.616088 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:42.619151 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:32:42.619124 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68955ce7_a707_49b0_8162_3ac3057c33da.slice/crio-a1b3c9970ccfce90d6b14f3e78fba584d71c11a9ce89c038fe80f6f103500afd WatchSource:0}: Error finding container a1b3c9970ccfce90d6b14f3e78fba584d71c11a9ce89c038fe80f6f103500afd: Status 404 returned error can't find the container with id a1b3c9970ccfce90d6b14f3e78fba584d71c11a9ce89c038fe80f6f103500afd Sep 19 12:32:42.683289 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.683110 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts"] Sep 19 12:32:42.686241 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:32:42.686213 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af75591_7757_494b_8957_e1fa3ddd0a4b.slice/crio-fde7d8421acedadf9f697c613d609f6a687f69ac29b91ef0f721ad9f69cf313f WatchSource:0}: Error finding container fde7d8421acedadf9f697c613d609f6a687f69ac29b91ef0f721ad9f69cf313f: Status 404 returned error can't find the container with id fde7d8421acedadf9f697c613d609f6a687f69ac29b91ef0f721ad9f69cf313f Sep 19 12:32:42.798669 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.798645 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p"] Sep 19 12:32:42.800244 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:32:42.800218 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b3a1cc_c0b6_4564_a92a_3dfaeb9e13dc.slice/crio-f7d6264d94ee3e596e135cc52f9df2e1ecb40afc493b61cd3342346107ab3b37 WatchSource:0}: Error finding container f7d6264d94ee3e596e135cc52f9df2e1ecb40afc493b61cd3342346107ab3b37: Status 404 returned error can't find the container with id f7d6264d94ee3e596e135cc52f9df2e1ecb40afc493b61cd3342346107ab3b37 Sep 19 12:32:42.886903 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.886878 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr"] Sep 19 12:32:42.898675 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.898641 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" event={"ID":"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc","Type":"ContainerStarted","Data":"c8ce490d2e1d85fe8bb0e3ec488523db6c4f4fcf64d6ee23de82a8a32753e52b"} Sep 19 12:32:42.898816 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.898709 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" event={"ID":"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc","Type":"ContainerStarted","Data":"f7d6264d94ee3e596e135cc52f9df2e1ecb40afc493b61cd3342346107ab3b37"} Sep 19 12:32:42.900226 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.900202 2585 generic.go:356] "Generic (PLEG): container finished" podID="68955ce7-a707-49b0-8162-3ac3057c33da" containerID="f864b07e851983563d19d73a826a0df6f6ab5fe8a5006dd5754aaa802273dbf9" exitCode=0 Sep 19 12:32:42.900357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.900277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" event={"ID":"68955ce7-a707-49b0-8162-3ac3057c33da","Type":"ContainerDied","Data":"f864b07e851983563d19d73a826a0df6f6ab5fe8a5006dd5754aaa802273dbf9"} Sep 19 12:32:42.900357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.900315 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" event={"ID":"68955ce7-a707-49b0-8162-3ac3057c33da","Type":"ContainerStarted","Data":"a1b3c9970ccfce90d6b14f3e78fba584d71c11a9ce89c038fe80f6f103500afd"} Sep 19 12:32:42.902707 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.902672 2585 generic.go:356] "Generic (PLEG): container finished" podID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerID="505f73064d6bf6ed09bb3b32a96b12c3b77c516ca9cd5ccd4d2b151f679aa9e1" exitCode=0 Sep 19 12:32:42.902798 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.902724 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" event={"ID":"1af75591-7757-494b-8957-e1fa3ddd0a4b","Type":"ContainerDied","Data":"505f73064d6bf6ed09bb3b32a96b12c3b77c516ca9cd5ccd4d2b151f679aa9e1"} Sep 19 12:32:42.902798 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:42.902783 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" event={"ID":"1af75591-7757-494b-8957-e1fa3ddd0a4b","Type":"ContainerStarted","Data":"fde7d8421acedadf9f697c613d609f6a687f69ac29b91ef0f721ad9f69cf313f"} Sep 19 12:32:42.928734 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:32:42.928706 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb639841f_ee52_4f1d_82c8_f0ccd0c769ae.slice/crio-a77018e76638298087b6d15ace46cd2e2fd9a51eebe08d2e62f6650f8c8f8e9e WatchSource:0}: Error finding container a77018e76638298087b6d15ace46cd2e2fd9a51eebe08d2e62f6650f8c8f8e9e: Status 404 returned error can't find the container with id a77018e76638298087b6d15ace46cd2e2fd9a51eebe08d2e62f6650f8c8f8e9e Sep 19 12:32:43.910088 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.910008 2585 generic.go:356] "Generic (PLEG): container finished" podID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerID="c8ce490d2e1d85fe8bb0e3ec488523db6c4f4fcf64d6ee23de82a8a32753e52b" exitCode=0 Sep 19 12:32:43.910498 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.910094 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" event={"ID":"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc","Type":"ContainerDied","Data":"c8ce490d2e1d85fe8bb0e3ec488523db6c4f4fcf64d6ee23de82a8a32753e52b"} Sep 19 12:32:43.911672 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.911654 2585 generic.go:356] "Generic (PLEG): container finished" podID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerID="554d2a80d916c5122a60252b86cf3cdcd6180629d4cc4d505d8309bfd70704c5" exitCode=0 Sep 19 12:32:43.911915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.911724 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" event={"ID":"b639841f-ee52-4f1d-82c8-f0ccd0c769ae","Type":"ContainerDied","Data":"554d2a80d916c5122a60252b86cf3cdcd6180629d4cc4d505d8309bfd70704c5"} Sep 19 12:32:43.911915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.911774 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" event={"ID":"b639841f-ee52-4f1d-82c8-f0ccd0c769ae","Type":"ContainerStarted","Data":"a77018e76638298087b6d15ace46cd2e2fd9a51eebe08d2e62f6650f8c8f8e9e"} Sep 19 12:32:43.913813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.913795 2585 generic.go:356] "Generic (PLEG): container finished" podID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerID="a75d67296ab2f00adb4c19b8623959fe953afbd60ab383ae604433b7e0f508f9" exitCode=0 Sep 19 12:32:43.913912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:43.913875 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" event={"ID":"1af75591-7757-494b-8957-e1fa3ddd0a4b","Type":"ContainerDied","Data":"a75d67296ab2f00adb4c19b8623959fe953afbd60ab383ae604433b7e0f508f9"} Sep 19 12:32:44.921759 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:44.921723 2585 generic.go:356] "Generic (PLEG): container finished" podID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerID="39f2c9897f5d79376aaed9a03cd5622fe3d23bcbfe3dcd2ed8a0fc17de5606c9" exitCode=0 Sep 19 12:32:44.922053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:44.921867 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" event={"ID":"1af75591-7757-494b-8957-e1fa3ddd0a4b","Type":"ContainerDied","Data":"39f2c9897f5d79376aaed9a03cd5622fe3d23bcbfe3dcd2ed8a0fc17de5606c9"} Sep 19 12:32:44.923793 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:44.923773 2585 generic.go:356] "Generic (PLEG): container finished" podID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerID="9df911095c232a349bb624c52f576a74352b25146bf4eeb1712584e30b0058e0" exitCode=0 Sep 19 12:32:44.923885 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:44.923863 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" event={"ID":"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc","Type":"ContainerDied","Data":"9df911095c232a349bb624c52f576a74352b25146bf4eeb1712584e30b0058e0"} Sep 19 12:32:44.925832 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:44.925812 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" event={"ID":"b639841f-ee52-4f1d-82c8-f0ccd0c769ae","Type":"ContainerStarted","Data":"508b4bd55b27a3204dcc102b9fa473110030835b23c7299f75a2b033070bea30"} Sep 19 12:32:45.934493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:45.934402 2585 generic.go:356] "Generic (PLEG): container finished" podID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerID="47f33c2ac86522bc2ac24c0a5ffdb3c3ebec7c51dbe4eebb5cc20632fe8e65d3" exitCode=0 Sep 19 12:32:45.934947 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:45.934487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" event={"ID":"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc","Type":"ContainerDied","Data":"47f33c2ac86522bc2ac24c0a5ffdb3c3ebec7c51dbe4eebb5cc20632fe8e65d3"} Sep 19 12:32:45.936315 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:45.936292 2585 generic.go:356] "Generic (PLEG): container finished" podID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerID="508b4bd55b27a3204dcc102b9fa473110030835b23c7299f75a2b033070bea30" exitCode=0 Sep 19 12:32:45.936422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:45.936335 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" event={"ID":"b639841f-ee52-4f1d-82c8-f0ccd0c769ae","Type":"ContainerDied","Data":"508b4bd55b27a3204dcc102b9fa473110030835b23c7299f75a2b033070bea30"} Sep 19 12:32:46.145065 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.145040 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:46.211549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.211478 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-util\") pod \"1af75591-7757-494b-8957-e1fa3ddd0a4b\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " Sep 19 12:32:46.211549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.211535 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-bundle\") pod \"1af75591-7757-494b-8957-e1fa3ddd0a4b\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " Sep 19 12:32:46.211799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.211616 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8sm6\" (UniqueName: \"kubernetes.io/projected/1af75591-7757-494b-8957-e1fa3ddd0a4b-kube-api-access-f8sm6\") pod \"1af75591-7757-494b-8957-e1fa3ddd0a4b\" (UID: \"1af75591-7757-494b-8957-e1fa3ddd0a4b\") " Sep 19 12:32:46.212202 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.212137 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-bundle" (OuterVolumeSpecName: "bundle") pod "1af75591-7757-494b-8957-e1fa3ddd0a4b" (UID: "1af75591-7757-494b-8957-e1fa3ddd0a4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:46.213736 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.213714 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af75591-7757-494b-8957-e1fa3ddd0a4b-kube-api-access-f8sm6" (OuterVolumeSpecName: "kube-api-access-f8sm6") pod "1af75591-7757-494b-8957-e1fa3ddd0a4b" (UID: "1af75591-7757-494b-8957-e1fa3ddd0a4b"). InnerVolumeSpecName "kube-api-access-f8sm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:46.216914 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.216881 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-util" (OuterVolumeSpecName: "util") pod "1af75591-7757-494b-8957-e1fa3ddd0a4b" (UID: "1af75591-7757-494b-8957-e1fa3ddd0a4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:46.313134 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.313106 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sm6\" (UniqueName: \"kubernetes.io/projected/1af75591-7757-494b-8957-e1fa3ddd0a4b-kube-api-access-f8sm6\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:46.313134 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.313131 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-util\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:46.313264 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.313141 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1af75591-7757-494b-8957-e1fa3ddd0a4b-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:46.944277 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.944236 2585 generic.go:356] "Generic (PLEG): container finished" podID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerID="7b8a529e776ad82e62140caeac78ddf2c7e121c652b58b47cc4e5667726d9349" exitCode=0 Sep 19 12:32:46.944661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.944314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" event={"ID":"b639841f-ee52-4f1d-82c8-f0ccd0c769ae","Type":"ContainerDied","Data":"7b8a529e776ad82e62140caeac78ddf2c7e121c652b58b47cc4e5667726d9349"} Sep 19 12:32:46.946433 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.946396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" event={"ID":"68955ce7-a707-49b0-8162-3ac3057c33da","Type":"ContainerStarted","Data":"5a38703a3b3a8e3198a9733ac222a996323cdd6c7b31cfaab41b58ed54e1528d"} Sep 19 12:32:46.948654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.948623 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" Sep 19 12:32:46.948654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.948643 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ard4ts" event={"ID":"1af75591-7757-494b-8957-e1fa3ddd0a4b","Type":"ContainerDied","Data":"fde7d8421acedadf9f697c613d609f6a687f69ac29b91ef0f721ad9f69cf313f"} Sep 19 12:32:46.948867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:46.948741 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde7d8421acedadf9f697c613d609f6a687f69ac29b91ef0f721ad9f69cf313f" Sep 19 12:32:47.085265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.085239 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:47.221242 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.221166 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-bundle\") pod \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " Sep 19 12:32:47.221242 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.221210 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-util\") pod \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " Sep 19 12:32:47.221419 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.221290 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpwgc\" (UniqueName: \"kubernetes.io/projected/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-kube-api-access-rpwgc\") pod \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\" (UID: \"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc\") " Sep 19 12:32:47.221704 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.221660 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-bundle" (OuterVolumeSpecName: "bundle") pod "f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" (UID: "f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:47.223512 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.223489 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-kube-api-access-rpwgc" (OuterVolumeSpecName: "kube-api-access-rpwgc") pod "f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" (UID: "f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc"). InnerVolumeSpecName "kube-api-access-rpwgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:47.226744 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.226711 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-util" (OuterVolumeSpecName: "util") pod "f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" (UID: "f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:47.322558 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.322527 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpwgc\" (UniqueName: \"kubernetes.io/projected/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-kube-api-access-rpwgc\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:47.322558 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.322554 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:47.322724 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.322567 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc-util\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:47.955564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.955490 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" Sep 19 12:32:47.955564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.955521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69wl5p" event={"ID":"f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc","Type":"ContainerDied","Data":"f7d6264d94ee3e596e135cc52f9df2e1ecb40afc493b61cd3342346107ab3b37"} Sep 19 12:32:47.956020 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.955565 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d6264d94ee3e596e135cc52f9df2e1ecb40afc493b61cd3342346107ab3b37" Sep 19 12:32:47.957346 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.957314 2585 generic.go:356] "Generic (PLEG): container finished" podID="68955ce7-a707-49b0-8162-3ac3057c33da" containerID="5a38703a3b3a8e3198a9733ac222a996323cdd6c7b31cfaab41b58ed54e1528d" exitCode=0 Sep 19 12:32:47.957434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:47.957391 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" event={"ID":"68955ce7-a707-49b0-8162-3ac3057c33da","Type":"ContainerDied","Data":"5a38703a3b3a8e3198a9733ac222a996323cdd6c7b31cfaab41b58ed54e1528d"} Sep 19 12:32:48.102600 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.102573 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:48.229156 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.229081 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-bundle\") pod \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " Sep 19 12:32:48.229156 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.229116 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-util\") pod \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " Sep 19 12:32:48.229156 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.229146 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8bl6\" (UniqueName: \"kubernetes.io/projected/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-kube-api-access-g8bl6\") pod \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\" (UID: \"b639841f-ee52-4f1d-82c8-f0ccd0c769ae\") " Sep 19 12:32:48.229623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.229598 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-bundle" (OuterVolumeSpecName: "bundle") pod "b639841f-ee52-4f1d-82c8-f0ccd0c769ae" (UID: "b639841f-ee52-4f1d-82c8-f0ccd0c769ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:48.231432 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.231412 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-kube-api-access-g8bl6" (OuterVolumeSpecName: "kube-api-access-g8bl6") pod "b639841f-ee52-4f1d-82c8-f0ccd0c769ae" (UID: "b639841f-ee52-4f1d-82c8-f0ccd0c769ae"). InnerVolumeSpecName "kube-api-access-g8bl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:48.234718 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.234673 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-util" (OuterVolumeSpecName: "util") pod "b639841f-ee52-4f1d-82c8-f0ccd0c769ae" (UID: "b639841f-ee52-4f1d-82c8-f0ccd0c769ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:48.330230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.330189 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:48.330230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.330224 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-util\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:48.330230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.330235 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bl6\" (UniqueName: \"kubernetes.io/projected/b639841f-ee52-4f1d-82c8-f0ccd0c769ae-kube-api-access-g8bl6\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:48.968417 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.968329 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" Sep 19 12:32:48.968872 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.968331 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72tbsr" event={"ID":"b639841f-ee52-4f1d-82c8-f0ccd0c769ae","Type":"ContainerDied","Data":"a77018e76638298087b6d15ace46cd2e2fd9a51eebe08d2e62f6650f8c8f8e9e"} Sep 19 12:32:48.968872 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.968453 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a77018e76638298087b6d15ace46cd2e2fd9a51eebe08d2e62f6650f8c8f8e9e" Sep 19 12:32:48.970827 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.970799 2585 generic.go:356] "Generic (PLEG): container finished" podID="68955ce7-a707-49b0-8162-3ac3057c33da" containerID="04717d99fa14e958e4b94090dd3ad46e87886f5262bc9d1944a400eecc17fa9a" exitCode=0 Sep 19 12:32:48.970920 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:48.970844 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" event={"ID":"68955ce7-a707-49b0-8162-3ac3057c33da","Type":"ContainerDied","Data":"04717d99fa14e958e4b94090dd3ad46e87886f5262bc9d1944a400eecc17fa9a"} Sep 19 12:32:50.109354 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.109320 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:50.247193 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.247126 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzmzx\" (UniqueName: \"kubernetes.io/projected/68955ce7-a707-49b0-8162-3ac3057c33da-kube-api-access-pzmzx\") pod \"68955ce7-a707-49b0-8162-3ac3057c33da\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " Sep 19 12:32:50.247317 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.247212 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-util\") pod \"68955ce7-a707-49b0-8162-3ac3057c33da\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " Sep 19 12:32:50.247317 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.247243 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-bundle\") pod \"68955ce7-a707-49b0-8162-3ac3057c33da\" (UID: \"68955ce7-a707-49b0-8162-3ac3057c33da\") " Sep 19 12:32:50.247792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.247765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-bundle" (OuterVolumeSpecName: "bundle") pod "68955ce7-a707-49b0-8162-3ac3057c33da" (UID: "68955ce7-a707-49b0-8162-3ac3057c33da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:50.249421 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.249392 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68955ce7-a707-49b0-8162-3ac3057c33da-kube-api-access-pzmzx" (OuterVolumeSpecName: "kube-api-access-pzmzx") pod "68955ce7-a707-49b0-8162-3ac3057c33da" (UID: "68955ce7-a707-49b0-8162-3ac3057c33da"). InnerVolumeSpecName "kube-api-access-pzmzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:32:50.252220 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.252176 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-util" (OuterVolumeSpecName: "util") pod "68955ce7-a707-49b0-8162-3ac3057c33da" (UID: "68955ce7-a707-49b0-8162-3ac3057c33da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:32:50.347929 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.347903 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzmzx\" (UniqueName: \"kubernetes.io/projected/68955ce7-a707-49b0-8162-3ac3057c33da-kube-api-access-pzmzx\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:50.347929 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.347926 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-util\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:50.348078 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.347935 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68955ce7-a707-49b0-8162-3ac3057c33da-bundle\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:32:50.984093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.984015 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" event={"ID":"68955ce7-a707-49b0-8162-3ac3057c33da","Type":"ContainerDied","Data":"a1b3c9970ccfce90d6b14f3e78fba584d71c11a9ce89c038fe80f6f103500afd"} Sep 19 12:32:50.984093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.984058 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b3c9970ccfce90d6b14f3e78fba584d71c11a9ce89c038fe80f6f103500afd" Sep 19 12:32:50.984093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:50.984067 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccbcl6" Sep 19 12:32:57.797411 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797379 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb"] Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797760 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="pull" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797772 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="pull" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797779 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="pull" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797784 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="pull" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797791 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797798 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797809 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="util" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797815 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="util" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797820 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797825 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797834 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797839 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797851 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="util" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797856 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="util" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797864 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="pull" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797869 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="pull" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797873 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="extract" Sep 19 12:32:57.797871 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797878 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="extract" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797886 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="util" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797891 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="util" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797897 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="util" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797901 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="util" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797907 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="pull" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797912 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="pull" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797969 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b3a1cc-c0b6-4564-a92a-3dfaeb9e13dc" containerName="extract" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797976 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b639841f-ee52-4f1d-82c8-f0ccd0c769ae" containerName="extract" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797983 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af75591-7757-494b-8957-e1fa3ddd0a4b" containerName="extract" Sep 19 12:32:57.798406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.797991 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="68955ce7-a707-49b0-8162-3ac3057c33da" containerName="extract" Sep 19 12:32:57.801261 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.801243 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:57.803872 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.803850 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 19 12:32:57.803992 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.803918 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tc7bx\"" Sep 19 12:32:57.803992 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.803950 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 19 12:32:57.813151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.813126 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb"] Sep 19 12:32:57.908542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.908504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/256eb4ab-ffff-454e-9b87-893ba57f0a3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:57.908676 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:57.908573 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkwx\" (UniqueName: \"kubernetes.io/projected/256eb4ab-ffff-454e-9b87-893ba57f0a3d-kube-api-access-fhkwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:58.009824 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:58.009791 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/256eb4ab-ffff-454e-9b87-893ba57f0a3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:58.009960 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:58.009889 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkwx\" (UniqueName: \"kubernetes.io/projected/256eb4ab-ffff-454e-9b87-893ba57f0a3d-kube-api-access-fhkwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:58.010130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:58.010111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/256eb4ab-ffff-454e-9b87-893ba57f0a3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:58.017865 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:58.017837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkwx\" (UniqueName: \"kubernetes.io/projected/256eb4ab-ffff-454e-9b87-893ba57f0a3d-kube-api-access-fhkwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:58.111880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:58.111827 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:32:58.391797 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:58.391774 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb"] Sep 19 12:32:58.393296 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:32:58.393269 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256eb4ab_ffff_454e_9b87_893ba57f0a3d.slice/crio-7280214a24d78cf45f8d61f43d83166ed2bf5376a38f04a5cb64013adbb7a97a WatchSource:0}: Error finding container 7280214a24d78cf45f8d61f43d83166ed2bf5376a38f04a5cb64013adbb7a97a: Status 404 returned error can't find the container with id 7280214a24d78cf45f8d61f43d83166ed2bf5376a38f04a5cb64013adbb7a97a Sep 19 12:32:59.029108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:59.029072 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" event={"ID":"256eb4ab-ffff-454e-9b87-893ba57f0a3d","Type":"ContainerStarted","Data":"7280214a24d78cf45f8d61f43d83166ed2bf5376a38f04a5cb64013adbb7a97a"} Sep 19 12:32:59.786414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:59.786377 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp"] Sep 19 12:32:59.790106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:59.790081 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:32:59.792521 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:59.792496 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7gmkc\"" Sep 19 12:32:59.801401 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:59.801376 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp"] Sep 19 12:32:59.928492 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:32:59.928453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5699p\" (UniqueName: \"kubernetes.io/projected/4dba0ea0-8941-46e1-83a1-5095f9a7852e-kube-api-access-5699p\") pod \"limitador-operator-controller-manager-74495c69f8-cvtwp\" (UID: \"4dba0ea0-8941-46e1-83a1-5095f9a7852e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:00.029894 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:00.029742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5699p\" (UniqueName: \"kubernetes.io/projected/4dba0ea0-8941-46e1-83a1-5095f9a7852e-kube-api-access-5699p\") pod \"limitador-operator-controller-manager-74495c69f8-cvtwp\" (UID: \"4dba0ea0-8941-46e1-83a1-5095f9a7852e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:00.039053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:00.038979 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5699p\" (UniqueName: \"kubernetes.io/projected/4dba0ea0-8941-46e1-83a1-5095f9a7852e-kube-api-access-5699p\") pod \"limitador-operator-controller-manager-74495c69f8-cvtwp\" (UID: \"4dba0ea0-8941-46e1-83a1-5095f9a7852e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:00.110746 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:00.110705 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:00.965511 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:00.965460 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp"] Sep 19 12:33:00.966304 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:00.966270 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dba0ea0_8941_46e1_83a1_5095f9a7852e.slice/crio-8fadbd97249ca56476fcb4518bb20750cd837bf772f07a4ba1bac09b981ee301 WatchSource:0}: Error finding container 8fadbd97249ca56476fcb4518bb20750cd837bf772f07a4ba1bac09b981ee301: Status 404 returned error can't find the container with id 8fadbd97249ca56476fcb4518bb20750cd837bf772f07a4ba1bac09b981ee301 Sep 19 12:33:01.045135 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:01.045097 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" event={"ID":"4dba0ea0-8941-46e1-83a1-5095f9a7852e","Type":"ContainerStarted","Data":"8fadbd97249ca56476fcb4518bb20750cd837bf772f07a4ba1bac09b981ee301"} Sep 19 12:33:02.095342 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.095309 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q"] Sep 19 12:33:02.099483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.099462 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:02.102237 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.102213 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tjvgz\"" Sep 19 12:33:02.102350 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.102280 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 19 12:33:02.107819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.107794 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q"] Sep 19 12:33:02.251923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.251891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/d1edbb71-d2c8-4484-a07d-eef385de2370-kube-api-access-n6mlq\") pod \"dns-operator-controller-manager-7b84cd7c9c-tbg7q\" (UID: \"d1edbb71-d2c8-4484-a07d-eef385de2370\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:02.353217 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.353128 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/d1edbb71-d2c8-4484-a07d-eef385de2370-kube-api-access-n6mlq\") pod \"dns-operator-controller-manager-7b84cd7c9c-tbg7q\" (UID: \"d1edbb71-d2c8-4484-a07d-eef385de2370\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:02.366275 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.366251 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mlq\" (UniqueName: \"kubernetes.io/projected/d1edbb71-d2c8-4484-a07d-eef385de2370-kube-api-access-n6mlq\") pod \"dns-operator-controller-manager-7b84cd7c9c-tbg7q\" (UID: \"d1edbb71-d2c8-4484-a07d-eef385de2370\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:02.413248 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:02.413217 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:03.826516 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:03.826487 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q"] Sep 19 12:33:03.827846 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:03.827818 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1edbb71_d2c8_4484_a07d_eef385de2370.slice/crio-eacba61c7c61a043863abb13f4887c23f05bd15c91c9e1bdcffd7a87cc96374b WatchSource:0}: Error finding container eacba61c7c61a043863abb13f4887c23f05bd15c91c9e1bdcffd7a87cc96374b: Status 404 returned error can't find the container with id eacba61c7c61a043863abb13f4887c23f05bd15c91c9e1bdcffd7a87cc96374b Sep 19 12:33:04.067125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.067088 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" event={"ID":"4dba0ea0-8941-46e1-83a1-5095f9a7852e","Type":"ContainerStarted","Data":"c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8"} Sep 19 12:33:04.067305 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.067256 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:04.069313 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.069243 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" event={"ID":"256eb4ab-ffff-454e-9b87-893ba57f0a3d","Type":"ContainerStarted","Data":"abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1"} Sep 19 12:33:04.069460 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.069370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:04.070783 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.070759 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" event={"ID":"d1edbb71-d2c8-4484-a07d-eef385de2370","Type":"ContainerStarted","Data":"eacba61c7c61a043863abb13f4887c23f05bd15c91c9e1bdcffd7a87cc96374b"} Sep 19 12:33:04.086436 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.086359 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" podStartSLOduration=2.522568428 podStartE2EDuration="5.086342421s" totalImagesPullingTime="2.563773993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:04.085227454 +0000 UTC m=+479.692078048" watchObservedRunningTime="2026-09-19 12:33:04.086342421 +0000 UTC m=+479.693192999" Sep 19 12:33:04.104327 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.104277 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" podStartSLOduration=1.964642964 podStartE2EDuration="7.104260731s" totalImagesPullingTime="5.139617767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:04.103789472 +0000 UTC m=+479.710640083" watchObservedRunningTime="2026-09-19 12:33:04.104260731 +0000 UTC m=+479.711111307" Sep 19 12:33:04.959659 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.959625 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bt64x"] Sep 19 12:33:04.964126 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.964102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:04.966868 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.966844 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f8tb2\"" Sep 19 12:33:04.972618 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:04.972574 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bt64x"] Sep 19 12:33:05.079487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:05.079455 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4nxf\" (UniqueName: \"kubernetes.io/projected/63b51a2b-429f-46e4-a243-6451235cf14a-kube-api-access-g4nxf\") pod \"authorino-operator-76b478bd78-bt64x\" (UID: \"63b51a2b-429f-46e4-a243-6451235cf14a\") " pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:05.180381 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:05.180341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4nxf\" (UniqueName: \"kubernetes.io/projected/63b51a2b-429f-46e4-a243-6451235cf14a-kube-api-access-g4nxf\") pod \"authorino-operator-76b478bd78-bt64x\" (UID: \"63b51a2b-429f-46e4-a243-6451235cf14a\") " pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:05.198272 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:05.198245 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4nxf\" (UniqueName: \"kubernetes.io/projected/63b51a2b-429f-46e4-a243-6451235cf14a-kube-api-access-g4nxf\") pod \"authorino-operator-76b478bd78-bt64x\" (UID: \"63b51a2b-429f-46e4-a243-6451235cf14a\") " pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:05.280838 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:05.280812 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:05.563667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:05.563637 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bt64x"] Sep 19 12:33:05.565370 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:05.565348 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b51a2b_429f_46e4_a243_6451235cf14a.slice/crio-6fdf9ee475c4c46b5ad5e01e0f143aa1d839dd274cbc5cb9457865c5c335bd4d WatchSource:0}: Error finding container 6fdf9ee475c4c46b5ad5e01e0f143aa1d839dd274cbc5cb9457865c5c335bd4d: Status 404 returned error can't find the container with id 6fdf9ee475c4c46b5ad5e01e0f143aa1d839dd274cbc5cb9457865c5c335bd4d Sep 19 12:33:06.087484 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:06.087435 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" event={"ID":"63b51a2b-429f-46e4-a243-6451235cf14a","Type":"ContainerStarted","Data":"6fdf9ee475c4c46b5ad5e01e0f143aa1d839dd274cbc5cb9457865c5c335bd4d"} Sep 19 12:33:06.089346 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:06.089312 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" event={"ID":"d1edbb71-d2c8-4484-a07d-eef385de2370","Type":"ContainerStarted","Data":"1192a7c0b7c2d3e3ff69ddaedf1694a22e11a64dae8d3de597499fb81e113b54"} Sep 19 12:33:06.089499 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:06.089453 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:06.113674 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:06.113618 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" podStartSLOduration=2.714277119 podStartE2EDuration="4.11359817s" totalImagesPullingTime="1.399321051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:06.111989838 +0000 UTC m=+481.718840445" watchObservedRunningTime="2026-09-19 12:33:06.11359817 +0000 UTC m=+481.720448753" Sep 19 12:33:08.103637 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:08.103598 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" event={"ID":"63b51a2b-429f-46e4-a243-6451235cf14a","Type":"ContainerStarted","Data":"c00b5dd60e9198f908e5931ba8931f18ba9ec9e9c07879b379a729a645fdd24e"} Sep 19 12:33:08.104016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:08.103675 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:08.119797 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:08.119755 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" podStartSLOduration=2.635885247 podStartE2EDuration="4.119742541s" totalImagesPullingTime="1.483857294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:08.118936067 +0000 UTC m=+483.725786661" watchObservedRunningTime="2026-09-19 12:33:08.119742541 +0000 UTC m=+483.726593111" Sep 19 12:33:15.082053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:15.082021 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:15.082437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:15.082089 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:16.692000 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.691971 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d"] Sep 19 12:33:16.695549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.695533 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:16.707550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.707526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d"] Sep 19 12:33:16.881111 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.881079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868d9760-6a6b-469a-b234-4cc944b3671c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:16.881265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.881143 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8tq\" (UniqueName: \"kubernetes.io/projected/868d9760-6a6b-469a-b234-4cc944b3671c-kube-api-access-lt8tq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:16.982196 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.982131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868d9760-6a6b-469a-b234-4cc944b3671c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:16.982196 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.982192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8tq\" (UniqueName: \"kubernetes.io/projected/868d9760-6a6b-469a-b234-4cc944b3671c-kube-api-access-lt8tq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:16.982561 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.982544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868d9760-6a6b-469a-b234-4cc944b3671c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:16.990137 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:16.990120 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8tq\" (UniqueName: \"kubernetes.io/projected/868d9760-6a6b-469a-b234-4cc944b3671c-kube-api-access-lt8tq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:17.005962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.005941 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:17.099018 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.098988 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tbg7q" Sep 19 12:33:17.128896 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.128868 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb"] Sep 19 12:33:17.129077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.129057 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" containerName="manager" containerID="cri-o://abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1" gracePeriod=2 Sep 19 12:33:17.131235 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.131199 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.136474 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.136452 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb"] Sep 19 12:33:17.145483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.145461 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d"] Sep 19 12:33:17.158498 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.158476 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d"] Sep 19 12:33:17.161545 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.161519 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7"] Sep 19 12:33:17.162162 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.162143 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" containerName="manager" Sep 19 12:33:17.162162 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.162163 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" containerName="manager" Sep 19 12:33:17.162328 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.162286 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" containerName="manager" Sep 19 12:33:17.165748 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.165671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.170082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.170061 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp"] Sep 19 12:33:17.170262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.170210 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" containerName="manager" containerID="cri-o://c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8" gracePeriod=2 Sep 19 12:33:17.174337 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.174316 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4"] Sep 19 12:33:17.178046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.178016 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7"] Sep 19 12:33:17.178153 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.178134 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.185989 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.185969 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp"] Sep 19 12:33:17.191510 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.191479 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.199354 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.199328 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4"] Sep 19 12:33:17.203505 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.203484 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q"] Sep 19 12:33:17.204132 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.204115 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" containerName="manager" Sep 19 12:33:17.204187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.204138 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" containerName="manager" Sep 19 12:33:17.204264 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.204248 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" containerName="manager" Sep 19 12:33:17.207565 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.207546 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:17.222567 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.222523 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.223420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.223395 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q"] Sep 19 12:33:17.224828 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.224797 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:17.227916 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.227892 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.229505 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.229483 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:17.284751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.284721 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdhk\" (UniqueName: \"kubernetes.io/projected/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-kube-api-access-mrdhk\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kwbt4\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.284867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.284848 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7xc\" (UniqueName: \"kubernetes.io/projected/505a1291-450a-4a0b-bc26-2aad6b5de43d-kube-api-access-cd7xc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8zfg7\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.284920 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.284882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505a1291-450a-4a0b-bc26-2aad6b5de43d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8zfg7\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.285028 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.285006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kwbt4\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.386166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4g5n\" (UniqueName: \"kubernetes.io/projected/5234da79-d945-482d-84c2-b8393a4e1223-kube-api-access-l4g5n\") pod \"limitador-operator-controller-manager-74495c69f8-8vx4q\" (UID: \"5234da79-d945-482d-84c2-b8393a4e1223\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:17.386166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386166 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdhk\" (UniqueName: \"kubernetes.io/projected/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-kube-api-access-mrdhk\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kwbt4\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.386368 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7xc\" (UniqueName: \"kubernetes.io/projected/505a1291-450a-4a0b-bc26-2aad6b5de43d-kube-api-access-cd7xc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8zfg7\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.386368 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505a1291-450a-4a0b-bc26-2aad6b5de43d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8zfg7\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.386487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kwbt4\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.386766 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kwbt4\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.386867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.386822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505a1291-450a-4a0b-bc26-2aad6b5de43d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8zfg7\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.395403 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.395275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdhk\" (UniqueName: \"kubernetes.io/projected/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-kube-api-access-mrdhk\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kwbt4\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.395531 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.395515 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7xc\" (UniqueName: \"kubernetes.io/projected/505a1291-450a-4a0b-bc26-2aad6b5de43d-kube-api-access-cd7xc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8zfg7\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.402261 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.402235 2585 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 19 12:33:17.402261 ip-10-0-131-149 kubenswrapper[2585]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-cgr5d_kuadrant-system_868d9760-6a6b-469a-b234-4cc944b3671c_0(c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-cgr5d to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d" Netns:"/var/run/netns/b019719d-5f45-46de-b82a-3b3980079249" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-cgr5d;K8S_POD_INFRA_CONTAINER_ID=c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d;K8S_POD_UID=868d9760-6a6b-469a-b234-4cc944b3671c" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d/868d9760-6a6b-469a-b234-4cc944b3671c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:30 [10.132.0.48/23] Sep 19 12:33:17.402261 ip-10-0-131-149 kubenswrapper[2585]: ' Sep 19 12:33:17.402261 ip-10-0-131-149 kubenswrapper[2585]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 19 12:33:17.402261 ip-10-0-131-149 kubenswrapper[2585]: > Sep 19 12:33:17.402548 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.402284 2585 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 19 12:33:17.402548 ip-10-0-131-149 kubenswrapper[2585]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-cgr5d_kuadrant-system_868d9760-6a6b-469a-b234-4cc944b3671c_0(c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-cgr5d to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d" Netns:"/var/run/netns/b019719d-5f45-46de-b82a-3b3980079249" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-cgr5d;K8S_POD_INFRA_CONTAINER_ID=c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d;K8S_POD_UID=868d9760-6a6b-469a-b234-4cc944b3671c" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d/868d9760-6a6b-469a-b234-4cc944b3671c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d c88b5e76c29b990f920294a27495b99df4793ae29d684d98b94940ee3d278a3d network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:30 [10.132.0.48/23] Sep 19 12:33:17.402548 ip-10-0-131-149 kubenswrapper[2585]: ' Sep 19 12:33:17.402548 ip-10-0-131-149 kubenswrapper[2585]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 19 12:33:17.402548 ip-10-0-131-149 kubenswrapper[2585]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:17.408093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.408072 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:17.410174 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.410146 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.411813 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.411790 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:17.414205 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.414187 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.416178 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.416158 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:17.417941 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:17.417919 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:17.487411 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.487381 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4g5n\" (UniqueName: \"kubernetes.io/projected/5234da79-d945-482d-84c2-b8393a4e1223-kube-api-access-l4g5n\") pod \"limitador-operator-controller-manager-74495c69f8-8vx4q\" (UID: \"5234da79-d945-482d-84c2-b8393a4e1223\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:17.497835 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.497817 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4g5n\" (UniqueName: \"kubernetes.io/projected/5234da79-d945-482d-84c2-b8393a4e1223-kube-api-access-l4g5n\") pod \"limitador-operator-controller-manager-74495c69f8-8vx4q\" (UID: \"5234da79-d945-482d-84c2-b8393a4e1223\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:17.560292 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.560258 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:17.567093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.567076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:17.574440 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.574419 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:17.588676 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.588580 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhkwx\" (UniqueName: \"kubernetes.io/projected/256eb4ab-ffff-454e-9b87-893ba57f0a3d-kube-api-access-fhkwx\") pod \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " Sep 19 12:33:17.588676 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.588640 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/256eb4ab-ffff-454e-9b87-893ba57f0a3d-extensions-socket-volume\") pod \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\" (UID: \"256eb4ab-ffff-454e-9b87-893ba57f0a3d\") " Sep 19 12:33:17.588842 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.588734 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5699p\" (UniqueName: \"kubernetes.io/projected/4dba0ea0-8941-46e1-83a1-5095f9a7852e-kube-api-access-5699p\") pod \"4dba0ea0-8941-46e1-83a1-5095f9a7852e\" (UID: \"4dba0ea0-8941-46e1-83a1-5095f9a7852e\") " Sep 19 12:33:17.589084 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.589056 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256eb4ab-ffff-454e-9b87-893ba57f0a3d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "256eb4ab-ffff-454e-9b87-893ba57f0a3d" (UID: "256eb4ab-ffff-454e-9b87-893ba57f0a3d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:33:17.591189 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.591149 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256eb4ab-ffff-454e-9b87-893ba57f0a3d-kube-api-access-fhkwx" (OuterVolumeSpecName: "kube-api-access-fhkwx") pod "256eb4ab-ffff-454e-9b87-893ba57f0a3d" (UID: "256eb4ab-ffff-454e-9b87-893ba57f0a3d"). InnerVolumeSpecName "kube-api-access-fhkwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:33:17.591629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.591609 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dba0ea0-8941-46e1-83a1-5095f9a7852e-kube-api-access-5699p" (OuterVolumeSpecName: "kube-api-access-5699p") pod "4dba0ea0-8941-46e1-83a1-5095f9a7852e" (UID: "4dba0ea0-8941-46e1-83a1-5095f9a7852e"). InnerVolumeSpecName "kube-api-access-5699p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:33:17.690302 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.690277 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhkwx\" (UniqueName: \"kubernetes.io/projected/256eb4ab-ffff-454e-9b87-893ba57f0a3d-kube-api-access-fhkwx\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:17.690418 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.690306 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/256eb4ab-ffff-454e-9b87-893ba57f0a3d-extensions-socket-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:17.690418 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.690320 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5699p\" (UniqueName: \"kubernetes.io/projected/4dba0ea0-8941-46e1-83a1-5095f9a7852e-kube-api-access-5699p\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:17.883790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.883575 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7"] Sep 19 12:33:17.886258 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.886234 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4"] Sep 19 12:33:17.887245 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:17.887210 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5234da79_d945_482d_84c2_b8393a4e1223.slice/crio-738cfa61c09e806566ec4d181b4b5acf34f4941bde1819c3cced09cbf656f297 WatchSource:0}: Error finding container 738cfa61c09e806566ec4d181b4b5acf34f4941bde1819c3cced09cbf656f297: Status 404 returned error can't find the container with id 738cfa61c09e806566ec4d181b4b5acf34f4941bde1819c3cced09cbf656f297 Sep 19 12:33:17.887872 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:17.887846 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebae533d_f9bb_4dd3_b00f_5d1be36ca20d.slice/crio-26a4624a8ad9e734e5bab0fcf68265210239a2ebb604cd7f6ac891bf756b292a WatchSource:0}: Error finding container 26a4624a8ad9e734e5bab0fcf68265210239a2ebb604cd7f6ac891bf756b292a: Status 404 returned error can't find the container with id 26a4624a8ad9e734e5bab0fcf68265210239a2ebb604cd7f6ac891bf756b292a Sep 19 12:33:17.888784 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:17.888761 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q"] Sep 19 12:33:17.889151 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:17.889133 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505a1291_450a_4a0b_bc26_2aad6b5de43d.slice/crio-6cfceae412e91ec3cda5a82554c066acb28673c6b7458c44608d9ff00b303bf9 WatchSource:0}: Error finding container 6cfceae412e91ec3cda5a82554c066acb28673c6b7458c44608d9ff00b303bf9: Status 404 returned error can't find the container with id 6cfceae412e91ec3cda5a82554c066acb28673c6b7458c44608d9ff00b303bf9 Sep 19 12:33:18.167166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.167077 2585 generic.go:356] "Generic (PLEG): container finished" podID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" containerID="c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8" exitCode=0 Sep 19 12:33:18.167166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.167128 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:18.167166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.167160 2585 scope.go:117] "RemoveContainer" containerID="c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8" Sep 19 12:33:18.170449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.169154 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" event={"ID":"5234da79-d945-482d-84c2-b8393a4e1223","Type":"ContainerStarted","Data":"c18e69410b96cb4285e5c4afd1a023a457e9ced3319363fdb98ced8e3cef07f6"} Sep 19 12:33:18.170449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.169207 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" event={"ID":"5234da79-d945-482d-84c2-b8393a4e1223","Type":"ContainerStarted","Data":"738cfa61c09e806566ec4d181b4b5acf34f4941bde1819c3cced09cbf656f297"} Sep 19 12:33:18.170449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.169828 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:18.170449 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.170017 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:18.172440 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.171974 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:18.172802 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.172769 2585 generic.go:356] "Generic (PLEG): container finished" podID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" containerID="abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1" exitCode=0 Sep 19 12:33:18.172999 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.172975 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:18.175827 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.175759 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:18.182713 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.180222 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:18.182713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.182618 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" event={"ID":"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d","Type":"ContainerStarted","Data":"d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474"} Sep 19 12:33:18.185462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.182907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" event={"ID":"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d","Type":"ContainerStarted","Data":"26a4624a8ad9e734e5bab0fcf68265210239a2ebb604cd7f6ac891bf756b292a"} Sep 19 12:33:18.185462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.183133 2585 scope.go:117] "RemoveContainer" containerID="c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8" Sep 19 12:33:18.185462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.183263 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:18.185462 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.183606 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8\": container with ID starting with c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8 not found: ID does not exist" containerID="c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8" Sep 19 12:33:18.185462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.183639 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8"} err="failed to get container status \"c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8\": rpc error: code = NotFound desc = could not find container \"c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8\": container with ID starting with c447ebf3b71adcf397f84636e3fd4d75cd0243519c4da03daa20e5944ae79ec8 not found: ID does not exist" Sep 19 12:33:18.185462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.183669 2585 scope.go:117] "RemoveContainer" containerID="abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1" Sep 19 12:33:18.186449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.186428 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:18.187177 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.187154 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" event={"ID":"505a1291-450a-4a0b-bc26-2aad6b5de43d","Type":"ContainerStarted","Data":"e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14"} Sep 19 12:33:18.187264 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.187213 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" event={"ID":"505a1291-450a-4a0b-bc26-2aad6b5de43d","Type":"ContainerStarted","Data":"6cfceae412e91ec3cda5a82554c066acb28673c6b7458c44608d9ff00b303bf9"} Sep 19 12:33:18.187385 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.187370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:18.196990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.196961 2585 scope.go:117] "RemoveContainer" containerID="abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1" Sep 19 12:33:18.197360 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.197336 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1\": container with ID starting with abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1 not found: ID does not exist" containerID="abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1" Sep 19 12:33:18.197418 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.197388 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1"} err="failed to get container status \"abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1\": rpc error: code = NotFound desc = could not find container \"abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1\": container with ID starting with abdc3c56faf4359c81c64bb6187c86f1e8b90b0fd7805b5a3cd62491b626e7d1 not found: ID does not exist" Sep 19 12:33:18.198110 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.198093 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:18.224178 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.224119 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" podStartSLOduration=1.224107219 podStartE2EDuration="1.224107219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:18.201471506 +0000 UTC m=+493.808322105" watchObservedRunningTime="2026-09-19 12:33:18.224107219 +0000 UTC m=+493.830957789" Sep 19 12:33:18.224278 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.224257 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="868d9760-6a6b-469a-b234-4cc944b3671c" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:18.224486 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.224456 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" podStartSLOduration=1.224449403 podStartE2EDuration="1.224449403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:18.222247134 +0000 UTC m=+493.829097727" watchObservedRunningTime="2026-09-19 12:33:18.224449403 +0000 UTC m=+493.831299982" Sep 19 12:33:18.264267 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.264236 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9kdtb\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kdtb" Sep 19 12:33:18.265379 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.265336 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" podStartSLOduration=1.265311408 podStartE2EDuration="1.265311408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:18.262236791 +0000 UTC m=+493.869087390" watchObservedRunningTime="2026-09-19 12:33:18.265311408 +0000 UTC m=+493.872161988" Sep 19 12:33:18.266668 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:18.266623 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cvtwp\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cvtwp" Sep 19 12:33:18.394862 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.394833 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868d9760-6a6b-469a-b234-4cc944b3671c-extensions-socket-volume\") pod \"868d9760-6a6b-469a-b234-4cc944b3671c\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " Sep 19 12:33:18.395030 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.394885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8tq\" (UniqueName: \"kubernetes.io/projected/868d9760-6a6b-469a-b234-4cc944b3671c-kube-api-access-lt8tq\") pod \"868d9760-6a6b-469a-b234-4cc944b3671c\" (UID: \"868d9760-6a6b-469a-b234-4cc944b3671c\") " Sep 19 12:33:18.395101 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.395040 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868d9760-6a6b-469a-b234-4cc944b3671c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "868d9760-6a6b-469a-b234-4cc944b3671c" (UID: "868d9760-6a6b-469a-b234-4cc944b3671c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:33:18.395178 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.395160 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868d9760-6a6b-469a-b234-4cc944b3671c-extensions-socket-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:18.397007 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.396986 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868d9760-6a6b-469a-b234-4cc944b3671c-kube-api-access-lt8tq" (OuterVolumeSpecName: "kube-api-access-lt8tq") pod "868d9760-6a6b-469a-b234-4cc944b3671c" (UID: "868d9760-6a6b-469a-b234-4cc944b3671c"). InnerVolumeSpecName "kube-api-access-lt8tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:33:18.495650 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.495571 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8tq\" (UniqueName: \"kubernetes.io/projected/868d9760-6a6b-469a-b234-4cc944b3671c-kube-api-access-lt8tq\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:18.579253 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.579222 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="256eb4ab-ffff-454e-9b87-893ba57f0a3d" path="/var/lib/kubelet/pods/256eb4ab-ffff-454e-9b87-893ba57f0a3d/volumes" Sep 19 12:33:18.579558 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.579544 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dba0ea0-8941-46e1-83a1-5095f9a7852e" path="/var/lib/kubelet/pods/4dba0ea0-8941-46e1-83a1-5095f9a7852e/volumes" Sep 19 12:33:18.579909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:18.579895 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="868d9760-6a6b-469a-b234-4cc944b3671c" path="/var/lib/kubelet/pods/868d9760-6a6b-469a-b234-4cc944b3671c/volumes" Sep 19 12:33:19.112397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:19.112368 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-bt64x" Sep 19 12:33:19.194783 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:19.194755 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:19.198666 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:19.198638 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-cgr5d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="868d9760-6a6b-469a-b234-4cc944b3671c" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-cgr5d" Sep 19 12:33:29.196724 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.196669 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:29.197163 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.196768 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:29.266414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.266377 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7"] Sep 19 12:33:29.266579 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.266557 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" podUID="505a1291-450a-4a0b-bc26-2aad6b5de43d" containerName="manager" containerID="cri-o://e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14" gracePeriod=10 Sep 19 12:33:29.524474 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.524450 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:29.579300 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.579275 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505a1291-450a-4a0b-bc26-2aad6b5de43d-extensions-socket-volume\") pod \"505a1291-450a-4a0b-bc26-2aad6b5de43d\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " Sep 19 12:33:29.579457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.579359 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd7xc\" (UniqueName: \"kubernetes.io/projected/505a1291-450a-4a0b-bc26-2aad6b5de43d-kube-api-access-cd7xc\") pod \"505a1291-450a-4a0b-bc26-2aad6b5de43d\" (UID: \"505a1291-450a-4a0b-bc26-2aad6b5de43d\") " Sep 19 12:33:29.579637 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.579614 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505a1291-450a-4a0b-bc26-2aad6b5de43d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "505a1291-450a-4a0b-bc26-2aad6b5de43d" (UID: "505a1291-450a-4a0b-bc26-2aad6b5de43d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:33:29.581397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.581374 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505a1291-450a-4a0b-bc26-2aad6b5de43d-kube-api-access-cd7xc" (OuterVolumeSpecName: "kube-api-access-cd7xc") pod "505a1291-450a-4a0b-bc26-2aad6b5de43d" (UID: "505a1291-450a-4a0b-bc26-2aad6b5de43d"). InnerVolumeSpecName "kube-api-access-cd7xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:33:29.615582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.615557 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54"] Sep 19 12:33:29.615981 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.615969 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505a1291-450a-4a0b-bc26-2aad6b5de43d" containerName="manager" Sep 19 12:33:29.616029 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.615984 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="505a1291-450a-4a0b-bc26-2aad6b5de43d" containerName="manager" Sep 19 12:33:29.616065 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.616061 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="505a1291-450a-4a0b-bc26-2aad6b5de43d" containerName="manager" Sep 19 12:33:29.620050 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.620037 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.634600 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.634578 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54"] Sep 19 12:33:29.680472 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.680446 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6p9v\" (UniqueName: \"kubernetes.io/projected/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-kube-api-access-k6p9v\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dhq54\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.680595 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.680514 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dhq54\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.680639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.680586 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd7xc\" (UniqueName: \"kubernetes.io/projected/505a1291-450a-4a0b-bc26-2aad6b5de43d-kube-api-access-cd7xc\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:29.680639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.680608 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505a1291-450a-4a0b-bc26-2aad6b5de43d-extensions-socket-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:29.781524 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.781465 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dhq54\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.781524 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.781509 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6p9v\" (UniqueName: \"kubernetes.io/projected/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-kube-api-access-k6p9v\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dhq54\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.781831 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.781814 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dhq54\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.791512 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.791494 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6p9v\" (UniqueName: \"kubernetes.io/projected/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-kube-api-access-k6p9v\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dhq54\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:29.930708 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:29.930657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:30.204471 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.204439 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8vx4q" Sep 19 12:33:30.209287 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.209262 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54"] Sep 19 12:33:30.212503 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:30.212473 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff3c7b8_2bfc_43df_97f9_3eff10ca07b4.slice/crio-00b04aa46f6c1e569229589bda3f94fe8ba7e11996eac215200c3c5acb0e9fa3 WatchSource:0}: Error finding container 00b04aa46f6c1e569229589bda3f94fe8ba7e11996eac215200c3c5acb0e9fa3: Status 404 returned error can't find the container with id 00b04aa46f6c1e569229589bda3f94fe8ba7e11996eac215200c3c5acb0e9fa3 Sep 19 12:33:30.266759 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.266728 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" event={"ID":"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4","Type":"ContainerStarted","Data":"00b04aa46f6c1e569229589bda3f94fe8ba7e11996eac215200c3c5acb0e9fa3"} Sep 19 12:33:30.268397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.268372 2585 generic.go:356] "Generic (PLEG): container finished" podID="505a1291-450a-4a0b-bc26-2aad6b5de43d" containerID="e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14" exitCode=0 Sep 19 12:33:30.268493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.268411 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" Sep 19 12:33:30.268493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.268458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" event={"ID":"505a1291-450a-4a0b-bc26-2aad6b5de43d","Type":"ContainerDied","Data":"e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14"} Sep 19 12:33:30.268608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.268503 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7" event={"ID":"505a1291-450a-4a0b-bc26-2aad6b5de43d","Type":"ContainerDied","Data":"6cfceae412e91ec3cda5a82554c066acb28673c6b7458c44608d9ff00b303bf9"} Sep 19 12:33:30.268608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.268532 2585 scope.go:117] "RemoveContainer" containerID="e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14" Sep 19 12:33:30.278523 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.278490 2585 scope.go:117] "RemoveContainer" containerID="e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14" Sep 19 12:33:30.278915 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:30.278831 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14\": container with ID starting with e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14 not found: ID does not exist" containerID="e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14" Sep 19 12:33:30.278915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.278878 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14"} err="failed to get container status \"e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14\": rpc error: code = NotFound desc = could not find container \"e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14\": container with ID starting with e28f506831b57d9089d0d6459e6bd0665806ab22b29c62d6cda1f561f9259b14 not found: ID does not exist" Sep 19 12:33:30.295267 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.295246 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7"] Sep 19 12:33:30.301664 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.301644 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8zfg7"] Sep 19 12:33:30.582943 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:30.582910 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505a1291-450a-4a0b-bc26-2aad6b5de43d" path="/var/lib/kubelet/pods/505a1291-450a-4a0b-bc26-2aad6b5de43d/volumes" Sep 19 12:33:31.278267 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:31.278233 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" event={"ID":"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4","Type":"ContainerStarted","Data":"ca0f381632d0760bf1e40a0de38b0f421fe4b2252316a1e93e722df1967ec161"} Sep 19 12:33:31.278663 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:31.278294 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:31.306150 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:31.306103 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" podStartSLOduration=2.306088769 podStartE2EDuration="2.306088769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:33:31.303938504 +0000 UTC m=+506.910789103" watchObservedRunningTime="2026-09-19 12:33:31.306088769 +0000 UTC m=+506.912939347" Sep 19 12:33:42.286834 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.286804 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:33:42.340476 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.340434 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4"] Sep 19 12:33:42.340717 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.340668 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" podUID="ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" containerName="manager" containerID="cri-o://d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474" gracePeriod=10 Sep 19 12:33:42.590494 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.590470 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:42.689320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.689286 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-extensions-socket-volume\") pod \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " Sep 19 12:33:42.689444 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.689335 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrdhk\" (UniqueName: \"kubernetes.io/projected/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-kube-api-access-mrdhk\") pod \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\" (UID: \"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d\") " Sep 19 12:33:42.689666 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.689644 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" (UID: "ebae533d-f9bb-4dd3-b00f-5d1be36ca20d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:33:42.691440 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.691420 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-kube-api-access-mrdhk" (OuterVolumeSpecName: "kube-api-access-mrdhk") pod "ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" (UID: "ebae533d-f9bb-4dd3-b00f-5d1be36ca20d"). InnerVolumeSpecName "kube-api-access-mrdhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:33:42.790104 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.790077 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-extensions-socket-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:42.790104 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:42.790102 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrdhk\" (UniqueName: \"kubernetes.io/projected/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d-kube-api-access-mrdhk\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:33:43.355798 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.355757 2585 generic.go:356] "Generic (PLEG): container finished" podID="ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" containerID="d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474" exitCode=0 Sep 19 12:33:43.356179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.355807 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" Sep 19 12:33:43.356179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.355842 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" event={"ID":"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d","Type":"ContainerDied","Data":"d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474"} Sep 19 12:33:43.356179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.355901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4" event={"ID":"ebae533d-f9bb-4dd3-b00f-5d1be36ca20d","Type":"ContainerDied","Data":"26a4624a8ad9e734e5bab0fcf68265210239a2ebb604cd7f6ac891bf756b292a"} Sep 19 12:33:43.356179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.355924 2585 scope.go:117] "RemoveContainer" containerID="d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474" Sep 19 12:33:43.367404 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.367368 2585 scope.go:117] "RemoveContainer" containerID="d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474" Sep 19 12:33:43.367849 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:33:43.367811 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474\": container with ID starting with d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474 not found: ID does not exist" containerID="d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474" Sep 19 12:33:43.367952 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.367865 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474"} err="failed to get container status \"d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474\": rpc error: code = NotFound desc = could not find container \"d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474\": container with ID starting with d39e5b886206265b0968f8db1e161b4456835d664bb239fc7903a3fd18cb9474 not found: ID does not exist" Sep 19 12:33:43.384770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.384743 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4"] Sep 19 12:33:43.388665 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:43.388646 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kwbt4"] Sep 19 12:33:44.578586 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:44.578551 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" path="/var/lib/kubelet/pods/ebae533d-f9bb-4dd3-b00f-5d1be36ca20d/volumes" Sep 19 12:33:57.809581 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.809499 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:33:57.809988 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.809964 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" containerName="manager" Sep 19 12:33:57.809988 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.809980 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" containerName="manager" Sep 19 12:33:57.810075 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.810064 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebae533d-f9bb-4dd3-b00f-5d1be36ca20d" containerName="manager" Sep 19 12:33:57.812166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.812149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:57.814553 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.814528 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 19 12:33:57.814675 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.814530 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v24kx\"" Sep 19 12:33:57.820564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.820540 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:33:57.907422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.907378 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:33:57.914986 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.914961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g88bq\" (UniqueName: \"kubernetes.io/projected/92d0dcdc-1efd-419e-843d-f5d840a77b68-kube-api-access-g88bq\") pod \"limitador-limitador-8588749977-xpgwq\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:57.915116 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:57.915011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92d0dcdc-1efd-419e-843d-f5d840a77b68-config-file\") pod \"limitador-limitador-8588749977-xpgwq\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:58.016257 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.016223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g88bq\" (UniqueName: \"kubernetes.io/projected/92d0dcdc-1efd-419e-843d-f5d840a77b68-kube-api-access-g88bq\") pod \"limitador-limitador-8588749977-xpgwq\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:58.016437 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.016294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92d0dcdc-1efd-419e-843d-f5d840a77b68-config-file\") pod \"limitador-limitador-8588749977-xpgwq\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:58.017006 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.016984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92d0dcdc-1efd-419e-843d-f5d840a77b68-config-file\") pod \"limitador-limitador-8588749977-xpgwq\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:58.023856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.023822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g88bq\" (UniqueName: \"kubernetes.io/projected/92d0dcdc-1efd-419e-843d-f5d840a77b68-kube-api-access-g88bq\") pod \"limitador-limitador-8588749977-xpgwq\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:58.123717 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.123636 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:33:58.401865 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.401839 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:33:58.403509 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:58.403481 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d0dcdc_1efd_419e_843d_f5d840a77b68.slice/crio-0ad1ec27fd0d934545d6436a6cdf277cf6206080b12821a58e4cfc43531d3ee0 WatchSource:0}: Error finding container 0ad1ec27fd0d934545d6436a6cdf277cf6206080b12821a58e4cfc43531d3ee0: Status 404 returned error can't find the container with id 0ad1ec27fd0d934545d6436a6cdf277cf6206080b12821a58e4cfc43531d3ee0 Sep 19 12:33:58.444708 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.444656 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" event={"ID":"92d0dcdc-1efd-419e-843d-f5d840a77b68","Type":"ContainerStarted","Data":"0ad1ec27fd0d934545d6436a6cdf277cf6206080b12821a58e4cfc43531d3ee0"} Sep 19 12:33:58.599817 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.599788 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqtbn"] Sep 19 12:33:58.603034 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.603020 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:33:58.605592 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.605572 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ll9kr\"" Sep 19 12:33:58.610451 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.610431 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqtbn"] Sep 19 12:33:58.723976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.723781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxt8\" (UniqueName: \"kubernetes.io/projected/9e4a56d8-0310-4ecc-ab84-417674eb983f-kube-api-access-rjxt8\") pod \"authorino-7bcfd8f9cd-wqtbn\" (UID: \"9e4a56d8-0310-4ecc-ab84-417674eb983f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:33:58.825208 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.825177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxt8\" (UniqueName: \"kubernetes.io/projected/9e4a56d8-0310-4ecc-ab84-417674eb983f-kube-api-access-rjxt8\") pod \"authorino-7bcfd8f9cd-wqtbn\" (UID: \"9e4a56d8-0310-4ecc-ab84-417674eb983f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:33:58.832905 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.832878 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxt8\" (UniqueName: \"kubernetes.io/projected/9e4a56d8-0310-4ecc-ab84-417674eb983f-kube-api-access-rjxt8\") pod \"authorino-7bcfd8f9cd-wqtbn\" (UID: \"9e4a56d8-0310-4ecc-ab84-417674eb983f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:33:58.913962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:58.913939 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:33:59.187339 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:59.187313 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqtbn"] Sep 19 12:33:59.189021 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:33:59.188993 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e4a56d8_0310_4ecc_ab84_417674eb983f.slice/crio-f9bd64ae1cd12b4ffbf1c2bf954eaa0d040424c956be6162469da0b251c5873c WatchSource:0}: Error finding container f9bd64ae1cd12b4ffbf1c2bf954eaa0d040424c956be6162469da0b251c5873c: Status 404 returned error can't find the container with id f9bd64ae1cd12b4ffbf1c2bf954eaa0d040424c956be6162469da0b251c5873c Sep 19 12:33:59.453206 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:33:59.453111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" event={"ID":"9e4a56d8-0310-4ecc-ab84-417674eb983f","Type":"ContainerStarted","Data":"f9bd64ae1cd12b4ffbf1c2bf954eaa0d040424c956be6162469da0b251c5873c"} Sep 19 12:34:00.461396 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:00.461342 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" event={"ID":"92d0dcdc-1efd-419e-843d-f5d840a77b68","Type":"ContainerStarted","Data":"17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806"} Sep 19 12:34:00.461920 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:00.461528 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:34:00.478544 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:00.478494 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" podStartSLOduration=2.378250569 podStartE2EDuration="3.478475113s" totalImagesPullingTime="1.100224544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:34:00.476582743 +0000 UTC m=+536.083433346" watchObservedRunningTime="2026-09-19 12:34:00.478475113 +0000 UTC m=+536.085325692" Sep 19 12:34:01.469289 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:01.469256 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" event={"ID":"9e4a56d8-0310-4ecc-ab84-417674eb983f","Type":"ContainerStarted","Data":"cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e"} Sep 19 12:34:01.485821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:01.485775 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" podStartSLOduration=1.865177047 podStartE2EDuration="3.485758099s" totalImagesPullingTime="1.620581052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:34:01.483416143 +0000 UTC m=+537.090266722" watchObservedRunningTime="2026-09-19 12:34:01.485758099 +0000 UTC m=+537.092608680" Sep 19 12:34:01.876669 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:01.876636 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqtbn"] Sep 19 12:34:03.481179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:03.481136 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" podUID="9e4a56d8-0310-4ecc-ab84-417674eb983f" containerName="authorino" containerID="cri-o://cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e" gracePeriod=30 Sep 19 12:34:03.731482 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:03.731425 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:34:03.871362 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:03.871329 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjxt8\" (UniqueName: \"kubernetes.io/projected/9e4a56d8-0310-4ecc-ab84-417674eb983f-kube-api-access-rjxt8\") pod \"9e4a56d8-0310-4ecc-ab84-417674eb983f\" (UID: \"9e4a56d8-0310-4ecc-ab84-417674eb983f\") " Sep 19 12:34:03.873386 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:03.873359 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4a56d8-0310-4ecc-ab84-417674eb983f-kube-api-access-rjxt8" (OuterVolumeSpecName: "kube-api-access-rjxt8") pod "9e4a56d8-0310-4ecc-ab84-417674eb983f" (UID: "9e4a56d8-0310-4ecc-ab84-417674eb983f"). InnerVolumeSpecName "kube-api-access-rjxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:34:03.971954 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:03.971931 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxt8\" (UniqueName: \"kubernetes.io/projected/9e4a56d8-0310-4ecc-ab84-417674eb983f-kube-api-access-rjxt8\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:34:04.487483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.487453 2585 generic.go:356] "Generic (PLEG): container finished" podID="9e4a56d8-0310-4ecc-ab84-417674eb983f" containerID="cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e" exitCode=0 Sep 19 12:34:04.487926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.487490 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" Sep 19 12:34:04.487926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.487535 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" event={"ID":"9e4a56d8-0310-4ecc-ab84-417674eb983f","Type":"ContainerDied","Data":"cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e"} Sep 19 12:34:04.487926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.487577 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wqtbn" event={"ID":"9e4a56d8-0310-4ecc-ab84-417674eb983f","Type":"ContainerDied","Data":"f9bd64ae1cd12b4ffbf1c2bf954eaa0d040424c956be6162469da0b251c5873c"} Sep 19 12:34:04.487926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.487595 2585 scope.go:117] "RemoveContainer" containerID="cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e" Sep 19 12:34:04.499199 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.499183 2585 scope.go:117] "RemoveContainer" containerID="cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e" Sep 19 12:34:04.499472 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:34:04.499451 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e\": container with ID starting with cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e not found: ID does not exist" containerID="cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e" Sep 19 12:34:04.499522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.499483 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e"} err="failed to get container status \"cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e\": rpc error: code = NotFound desc = could not find container \"cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e\": container with ID starting with cd365346d77722c739c2c3f8931c8e77547f75703e755b767bf94b316340571e not found: ID does not exist" Sep 19 12:34:04.511228 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.511204 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqtbn"] Sep 19 12:34:04.514994 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.514973 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wqtbn"] Sep 19 12:34:04.579083 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:04.579060 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e4a56d8-0310-4ecc-ab84-417674eb983f" path="/var/lib/kubelet/pods/9e4a56d8-0310-4ecc-ab84-417674eb983f/volumes" Sep 19 12:34:11.470195 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:11.470165 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:34:12.875968 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:12.875934 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:34:12.876360 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:12.876117 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" podUID="92d0dcdc-1efd-419e-843d-f5d840a77b68" containerName="limitador" containerID="cri-o://17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806" gracePeriod=30 Sep 19 12:34:13.428654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.428632 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:34:13.545505 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.545473 2585 generic.go:356] "Generic (PLEG): container finished" podID="92d0dcdc-1efd-419e-843d-f5d840a77b68" containerID="17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806" exitCode=0 Sep 19 12:34:13.545658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.545517 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" Sep 19 12:34:13.545658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.545561 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" event={"ID":"92d0dcdc-1efd-419e-843d-f5d840a77b68","Type":"ContainerDied","Data":"17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806"} Sep 19 12:34:13.545658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.545619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpgwq" event={"ID":"92d0dcdc-1efd-419e-843d-f5d840a77b68","Type":"ContainerDied","Data":"0ad1ec27fd0d934545d6436a6cdf277cf6206080b12821a58e4cfc43531d3ee0"} Sep 19 12:34:13.545658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.545647 2585 scope.go:117] "RemoveContainer" containerID="17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806" Sep 19 12:34:13.546189 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.546172 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92d0dcdc-1efd-419e-843d-f5d840a77b68-config-file\") pod \"92d0dcdc-1efd-419e-843d-f5d840a77b68\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " Sep 19 12:34:13.546303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.546288 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g88bq\" (UniqueName: \"kubernetes.io/projected/92d0dcdc-1efd-419e-843d-f5d840a77b68-kube-api-access-g88bq\") pod \"92d0dcdc-1efd-419e-843d-f5d840a77b68\" (UID: \"92d0dcdc-1efd-419e-843d-f5d840a77b68\") " Sep 19 12:34:13.546518 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.546495 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d0dcdc-1efd-419e-843d-f5d840a77b68-config-file" (OuterVolumeSpecName: "config-file") pod "92d0dcdc-1efd-419e-843d-f5d840a77b68" (UID: "92d0dcdc-1efd-419e-843d-f5d840a77b68"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:34:13.548520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.548491 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d0dcdc-1efd-419e-843d-f5d840a77b68-kube-api-access-g88bq" (OuterVolumeSpecName: "kube-api-access-g88bq") pod "92d0dcdc-1efd-419e-843d-f5d840a77b68" (UID: "92d0dcdc-1efd-419e-843d-f5d840a77b68"). InnerVolumeSpecName "kube-api-access-g88bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:34:13.560535 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.560519 2585 scope.go:117] "RemoveContainer" containerID="17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806" Sep 19 12:34:13.560857 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:34:13.560840 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806\": container with ID starting with 17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806 not found: ID does not exist" containerID="17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806" Sep 19 12:34:13.560899 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.560866 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806"} err="failed to get container status \"17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806\": rpc error: code = NotFound desc = could not find container \"17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806\": container with ID starting with 17aaa7bd50d295897d918cdb59196ca0cfe43aceb55518840e431ff674a8c806 not found: ID does not exist" Sep 19 12:34:13.646997 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.646974 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g88bq\" (UniqueName: \"kubernetes.io/projected/92d0dcdc-1efd-419e-843d-f5d840a77b68-kube-api-access-g88bq\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:34:13.646997 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.646996 2585 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92d0dcdc-1efd-419e-843d-f5d840a77b68-config-file\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:34:13.869262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.869229 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:34:13.872847 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:13.872825 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpgwq"] Sep 19 12:34:14.441749 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.441715 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs"] Sep 19 12:34:14.442469 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.442447 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d0dcdc-1efd-419e-843d-f5d840a77b68" containerName="limitador" Sep 19 12:34:14.442519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.442475 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d0dcdc-1efd-419e-843d-f5d840a77b68" containerName="limitador" Sep 19 12:34:14.442519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.442487 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4a56d8-0310-4ecc-ab84-417674eb983f" containerName="authorino" Sep 19 12:34:14.442519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.442495 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4a56d8-0310-4ecc-ab84-417674eb983f" containerName="authorino" Sep 19 12:34:14.442612 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.442602 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e4a56d8-0310-4ecc-ab84-417674eb983f" containerName="authorino" Sep 19 12:34:14.442646 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.442617 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d0dcdc-1efd-419e-843d-f5d840a77b68" containerName="limitador" Sep 19 12:34:14.447630 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.447605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.450052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.450026 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 19 12:34:14.450199 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.450100 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 19 12:34:14.450799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.450781 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 12:34:14.450969 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.450803 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zxv7j\"" Sep 19 12:34:14.451210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.451178 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs"] Sep 19 12:34:14.555624 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.555596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4hn\" (UniqueName: \"kubernetes.io/projected/4ae9128a-239b-465d-a853-b6adcd87fab3-kube-api-access-nd4hn\") pod \"postgres-69fb4594cf-lzkxs\" (UID: \"4ae9128a-239b-465d-a853-b6adcd87fab3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.555786 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.555629 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ae9128a-239b-465d-a853-b6adcd87fab3-data\") pod \"postgres-69fb4594cf-lzkxs\" (UID: \"4ae9128a-239b-465d-a853-b6adcd87fab3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.579666 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.579637 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d0dcdc-1efd-419e-843d-f5d840a77b68" path="/var/lib/kubelet/pods/92d0dcdc-1efd-419e-843d-f5d840a77b68/volumes" Sep 19 12:34:14.656372 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.656347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4hn\" (UniqueName: \"kubernetes.io/projected/4ae9128a-239b-465d-a853-b6adcd87fab3-kube-api-access-nd4hn\") pod \"postgres-69fb4594cf-lzkxs\" (UID: \"4ae9128a-239b-465d-a853-b6adcd87fab3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.656488 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.656377 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ae9128a-239b-465d-a853-b6adcd87fab3-data\") pod \"postgres-69fb4594cf-lzkxs\" (UID: \"4ae9128a-239b-465d-a853-b6adcd87fab3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.656788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.656768 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ae9128a-239b-465d-a853-b6adcd87fab3-data\") pod \"postgres-69fb4594cf-lzkxs\" (UID: \"4ae9128a-239b-465d-a853-b6adcd87fab3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.665976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.665953 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4hn\" (UniqueName: \"kubernetes.io/projected/4ae9128a-239b-465d-a853-b6adcd87fab3-kube-api-access-nd4hn\") pod \"postgres-69fb4594cf-lzkxs\" (UID: \"4ae9128a-239b-465d-a853-b6adcd87fab3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:14.759250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:14.759183 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:15.033487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:15.033461 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs"] Sep 19 12:34:15.034626 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:34:15.034593 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae9128a_239b_465d_a853_b6adcd87fab3.slice/crio-fab4eb92e1fa2354f0546eb7567c4a30f1357ad55fcc7cfb47579c1a5eb7e261 WatchSource:0}: Error finding container fab4eb92e1fa2354f0546eb7567c4a30f1357ad55fcc7cfb47579c1a5eb7e261: Status 404 returned error can't find the container with id fab4eb92e1fa2354f0546eb7567c4a30f1357ad55fcc7cfb47579c1a5eb7e261 Sep 19 12:34:15.559486 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:15.559452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" event={"ID":"4ae9128a-239b-465d-a853-b6adcd87fab3","Type":"ContainerStarted","Data":"fab4eb92e1fa2354f0546eb7567c4a30f1357ad55fcc7cfb47579c1a5eb7e261"} Sep 19 12:34:20.596845 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:20.596804 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" event={"ID":"4ae9128a-239b-465d-a853-b6adcd87fab3","Type":"ContainerStarted","Data":"f2b87ad080ef5e0d1bfa160baf351598561ba84b87aa662b35bf841cb1eca26b"} Sep 19 12:34:20.597233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:20.596866 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:20.613522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:20.613476 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" podStartSLOduration=1.680241161 podStartE2EDuration="6.613456128s" totalImagesPullingTime="4.933214967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:34:20.610323228 +0000 UTC m=+556.217173827" watchObservedRunningTime="2026-09-19 12:34:20.613456128 +0000 UTC m=+556.220306707" Sep 19 12:34:26.633177 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:26.633150 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzkxs" Sep 19 12:34:27.429309 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.429277 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d87c84dd6-nqts2"] Sep 19 12:34:27.435565 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.435543 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.438581 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.438557 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ll9kr\"" Sep 19 12:34:27.438767 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.438561 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 19 12:34:27.439722 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.439666 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d87c84dd6-nqts2"] Sep 19 12:34:27.464049 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.464020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/867e3b4a-baa2-44d6-ac98-193cb7d2740f-tls-cert\") pod \"authorino-7d87c84dd6-nqts2\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.464152 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.464050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpw9\" (UniqueName: \"kubernetes.io/projected/867e3b4a-baa2-44d6-ac98-193cb7d2740f-kube-api-access-kvpw9\") pod \"authorino-7d87c84dd6-nqts2\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.564551 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.564524 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/867e3b4a-baa2-44d6-ac98-193cb7d2740f-tls-cert\") pod \"authorino-7d87c84dd6-nqts2\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.564730 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.564566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpw9\" (UniqueName: \"kubernetes.io/projected/867e3b4a-baa2-44d6-ac98-193cb7d2740f-kube-api-access-kvpw9\") pod \"authorino-7d87c84dd6-nqts2\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.567100 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.567074 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/867e3b4a-baa2-44d6-ac98-193cb7d2740f-tls-cert\") pod \"authorino-7d87c84dd6-nqts2\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.572757 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.572737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpw9\" (UniqueName: \"kubernetes.io/projected/867e3b4a-baa2-44d6-ac98-193cb7d2740f-kube-api-access-kvpw9\") pod \"authorino-7d87c84dd6-nqts2\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:27.747609 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:27.747533 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:34:28.018051 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:28.018026 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d87c84dd6-nqts2"] Sep 19 12:34:28.019996 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:34:28.019973 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod867e3b4a_baa2_44d6_ac98_193cb7d2740f.slice/crio-0ec9f30256afcc1f493f56e000f44acd181c9988bf88388638025504a24ed0b5 WatchSource:0}: Error finding container 0ec9f30256afcc1f493f56e000f44acd181c9988bf88388638025504a24ed0b5: Status 404 returned error can't find the container with id 0ec9f30256afcc1f493f56e000f44acd181c9988bf88388638025504a24ed0b5 Sep 19 12:34:28.645564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:28.645525 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" event={"ID":"867e3b4a-baa2-44d6-ac98-193cb7d2740f","Type":"ContainerStarted","Data":"da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25"} Sep 19 12:34:28.645725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:28.645571 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" event={"ID":"867e3b4a-baa2-44d6-ac98-193cb7d2740f","Type":"ContainerStarted","Data":"0ec9f30256afcc1f493f56e000f44acd181c9988bf88388638025504a24ed0b5"} Sep 19 12:34:28.662174 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:28.662128 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" podStartSLOduration=1.266753985 podStartE2EDuration="1.662110738s" totalImagesPullingTime="395.356753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:34:28.659985913 +0000 UTC m=+564.266836491" watchObservedRunningTime="2026-09-19 12:34:28.662110738 +0000 UTC m=+564.268961319" Sep 19 12:34:33.265082 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.265046 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-d44tl"] Sep 19 12:34:33.283895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.283870 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-d44tl"] Sep 19 12:34:33.284033 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.283963 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.286413 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.286387 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 19 12:34:33.286599 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.286419 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 19 12:34:33.286711 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.286463 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhllr\"" Sep 19 12:34:33.286823 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.286478 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 19 12:34:33.315610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.315588 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-webhook-cert\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.315768 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.315624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.315768 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.315672 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gql\" (UniqueName: \"kubernetes.io/projected/9987eb99-4c11-4229-b48b-d7fb17669edd-kube-api-access-v5gql\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.416962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.416937 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-webhook-cert\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.417100 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.416971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.417100 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.417015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gql\" (UniqueName: \"kubernetes.io/projected/9987eb99-4c11-4229-b48b-d7fb17669edd-kube-api-access-v5gql\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.417178 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:34:33.417152 2585 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 19 12:34:33.417227 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:34:33.417215 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs podName:9987eb99-4c11-4229-b48b-d7fb17669edd nodeName:}" failed. No retries permitted until 2026-09-19 12:34:33.917196625 +0000 UTC m=+569.524047188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs") pod "maas-controller-b7fb6b9f7-d44tl" (UID: "9987eb99-4c11-4229-b48b-d7fb17669edd") : secret "maas-controller-metrics-tls" not found Sep 19 12:34:33.419536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.419513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-webhook-cert\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.424782 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.424759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gql\" (UniqueName: \"kubernetes.io/projected/9987eb99-4c11-4229-b48b-d7fb17669edd-kube-api-access-v5gql\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.443179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.443157 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c69b6d687-rc65l"] Sep 19 12:34:33.445539 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.445525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.455192 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.455170 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-rc65l"] Sep 19 12:34:33.518060 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.518002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-metrics-certs\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.518060 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.518053 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-webhook-cert\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.518203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.518074 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6gt\" (UniqueName: \"kubernetes.io/projected/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-kube-api-access-jr6gt\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.619003 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.618980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-webhook-cert\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.619097 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.619008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6gt\" (UniqueName: \"kubernetes.io/projected/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-kube-api-access-jr6gt\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.619097 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.619071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-metrics-certs\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.621667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.621639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-metrics-certs\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.621771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.621716 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-webhook-cert\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.626403 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.626385 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6gt\" (UniqueName: \"kubernetes.io/projected/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-kube-api-access-jr6gt\") pod \"maas-controller-6c69b6d687-rc65l\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.757127 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.757101 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:33.923138 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.923101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:33.925527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:33.925508 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs\") pod \"maas-controller-b7fb6b9f7-d44tl\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:34.027639 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:34.027616 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-rc65l"] Sep 19 12:34:34.028544 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:34:34.028523 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1106c14_d2cb_4dce_8c8a_49c5187f0c56.slice/crio-a35fdade2d1b302506a3f7eff2eead2753e5548dfe814392663d2146969fb6c3 WatchSource:0}: Error finding container a35fdade2d1b302506a3f7eff2eead2753e5548dfe814392663d2146969fb6c3: Status 404 returned error can't find the container with id a35fdade2d1b302506a3f7eff2eead2753e5548dfe814392663d2146969fb6c3 Sep 19 12:34:34.195988 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:34.195929 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:34.468972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:34.468943 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-d44tl"] Sep 19 12:34:34.470066 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:34:34.470042 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9987eb99_4c11_4229_b48b_d7fb17669edd.slice/crio-fc5460d8c38018b54c2129243356ccee806e074790c7c6d8b241d50764939eef WatchSource:0}: Error finding container fc5460d8c38018b54c2129243356ccee806e074790c7c6d8b241d50764939eef: Status 404 returned error can't find the container with id fc5460d8c38018b54c2129243356ccee806e074790c7c6d8b241d50764939eef Sep 19 12:34:34.684563 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:34.684521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-rc65l" event={"ID":"e1106c14-d2cb-4dce-8c8a-49c5187f0c56","Type":"ContainerStarted","Data":"a35fdade2d1b302506a3f7eff2eead2753e5548dfe814392663d2146969fb6c3"} Sep 19 12:34:34.685986 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:34.685961 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" event={"ID":"9987eb99-4c11-4229-b48b-d7fb17669edd","Type":"ContainerStarted","Data":"fc5460d8c38018b54c2129243356ccee806e074790c7c6d8b241d50764939eef"} Sep 19 12:34:37.708047 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:37.708011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" event={"ID":"9987eb99-4c11-4229-b48b-d7fb17669edd","Type":"ContainerStarted","Data":"151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93"} Sep 19 12:34:37.708501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:37.708074 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:37.709505 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:37.709480 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-rc65l" event={"ID":"e1106c14-d2cb-4dce-8c8a-49c5187f0c56","Type":"ContainerStarted","Data":"f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb"} Sep 19 12:34:37.709596 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:37.709562 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:37.726534 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:37.726487 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" podStartSLOduration=2.526431939 podStartE2EDuration="4.726473546s" totalImagesPullingTime="2.200041607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:34:37.724314177 +0000 UTC m=+573.331164775" watchObservedRunningTime="2026-09-19 12:34:37.726473546 +0000 UTC m=+573.333324125" Sep 19 12:34:37.742118 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:37.742077 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c69b6d687-rc65l" podStartSLOduration=2.112514408 podStartE2EDuration="4.742063409s" totalImagesPullingTime="2.629549001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:34:37.741042586 +0000 UTC m=+573.347893188" watchObservedRunningTime="2026-09-19 12:34:37.742063409 +0000 UTC m=+573.348913988" Sep 19 12:34:48.717118 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:48.717087 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 12:34:48.717501 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:48.717259 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:48.779559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:48.779530 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-d44tl"] Sep 19 12:34:48.779855 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:48.779717 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" podUID="9987eb99-4c11-4229-b48b-d7fb17669edd" containerName="manager" containerID="cri-o://151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93" gracePeriod=30 Sep 19 12:34:49.032392 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.032367 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:49.155644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.155611 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-webhook-cert\") pod \"9987eb99-4c11-4229-b48b-d7fb17669edd\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " Sep 19 12:34:49.155839 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.155650 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs\") pod \"9987eb99-4c11-4229-b48b-d7fb17669edd\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " Sep 19 12:34:49.155839 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.155753 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5gql\" (UniqueName: \"kubernetes.io/projected/9987eb99-4c11-4229-b48b-d7fb17669edd-kube-api-access-v5gql\") pod \"9987eb99-4c11-4229-b48b-d7fb17669edd\" (UID: \"9987eb99-4c11-4229-b48b-d7fb17669edd\") " Sep 19 12:34:49.158001 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.157975 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9987eb99-4c11-4229-b48b-d7fb17669edd" (UID: "9987eb99-4c11-4229-b48b-d7fb17669edd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:34:49.158106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.158036 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9987eb99-4c11-4229-b48b-d7fb17669edd-kube-api-access-v5gql" (OuterVolumeSpecName: "kube-api-access-v5gql") pod "9987eb99-4c11-4229-b48b-d7fb17669edd" (UID: "9987eb99-4c11-4229-b48b-d7fb17669edd"). InnerVolumeSpecName "kube-api-access-v5gql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:34:49.158106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.158040 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9987eb99-4c11-4229-b48b-d7fb17669edd" (UID: "9987eb99-4c11-4229-b48b-d7fb17669edd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:34:49.256410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.256388 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-webhook-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:34:49.256410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.256408 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9987eb99-4c11-4229-b48b-d7fb17669edd-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:34:49.256545 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.256417 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gql\" (UniqueName: \"kubernetes.io/projected/9987eb99-4c11-4229-b48b-d7fb17669edd-kube-api-access-v5gql\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:34:49.786108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.786072 2585 generic.go:356] "Generic (PLEG): container finished" podID="9987eb99-4c11-4229-b48b-d7fb17669edd" containerID="151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93" exitCode=0 Sep 19 12:34:49.786491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.786117 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" Sep 19 12:34:49.786491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.786115 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" event={"ID":"9987eb99-4c11-4229-b48b-d7fb17669edd","Type":"ContainerDied","Data":"151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93"} Sep 19 12:34:49.786491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.786239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-d44tl" event={"ID":"9987eb99-4c11-4229-b48b-d7fb17669edd","Type":"ContainerDied","Data":"fc5460d8c38018b54c2129243356ccee806e074790c7c6d8b241d50764939eef"} Sep 19 12:34:49.786491 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.786267 2585 scope.go:117] "RemoveContainer" containerID="151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93" Sep 19 12:34:49.796295 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.796279 2585 scope.go:117] "RemoveContainer" containerID="151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93" Sep 19 12:34:49.796571 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:34:49.796550 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93\": container with ID starting with 151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93 not found: ID does not exist" containerID="151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93" Sep 19 12:34:49.796717 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.796578 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93"} err="failed to get container status \"151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93\": rpc error: code = NotFound desc = could not find container \"151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93\": container with ID starting with 151bb16ba3f1f0edb644f721375f1a03b12f35d5ed8d22aeed28d43653685f93 not found: ID does not exist" Sep 19 12:34:49.813206 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.813177 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-d44tl"] Sep 19 12:34:49.813374 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:49.813361 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-d44tl"] Sep 19 12:34:50.579324 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:34:50.579295 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9987eb99-4c11-4229-b48b-d7fb17669edd" path="/var/lib/kubelet/pods/9987eb99-4c11-4229-b48b-d7fb17669edd/volumes" Sep 19 12:35:06.488253 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:06.488220 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:35:06.492906 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:06.492885 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:35:07.834424 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.834392 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5"] Sep 19 12:35:07.834939 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.834922 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9987eb99-4c11-4229-b48b-d7fb17669edd" containerName="manager" Sep 19 12:35:07.834989 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.834942 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9987eb99-4c11-4229-b48b-d7fb17669edd" containerName="manager" Sep 19 12:35:07.835061 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.835044 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9987eb99-4c11-4229-b48b-d7fb17669edd" containerName="manager" Sep 19 12:35:07.839624 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.839607 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:07.841900 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.841871 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 19 12:35:07.842017 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.841963 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 19 12:35:07.849303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:07.849280 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5"] Sep 19 12:35:08.020709 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.020662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-metrics-certs\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.020866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.020787 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-maas-api-tls\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.020866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.020824 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9qz\" (UniqueName: \"kubernetes.io/projected/a67b123f-64f5-40c1-920c-2bdea84b5b7c-kube-api-access-kz9qz\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.020866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.020858 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67b123f-64f5-40c1-920c-2bdea84b5b7c-openshift-service-ca\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.122448 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.122368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-metrics-certs\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.122613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.122467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-maas-api-tls\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.122613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.122514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9qz\" (UniqueName: \"kubernetes.io/projected/a67b123f-64f5-40c1-920c-2bdea84b5b7c-kube-api-access-kz9qz\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.122613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.122558 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67b123f-64f5-40c1-920c-2bdea84b5b7c-openshift-service-ca\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.123584 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.123558 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67b123f-64f5-40c1-920c-2bdea84b5b7c-openshift-service-ca\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.125142 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.125115 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-maas-api-tls\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.125542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.125516 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-metrics-certs\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.130367 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.130347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9qz\" (UniqueName: \"kubernetes.io/projected/a67b123f-64f5-40c1-920c-2bdea84b5b7c-kube-api-access-kz9qz\") pod \"maas-api-5598d8c9fb-vw4v5\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.151922 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.151901 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:08.374222 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.374150 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx"] Sep 19 12:35:08.381545 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.381523 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.385239 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.384936 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vvzvj\"" Sep 19 12:35:08.385239 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.385136 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 19 12:35:08.387386 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.387361 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx"] Sep 19 12:35:08.441055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.441024 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5"] Sep 19 12:35:08.442045 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:35:08.442020 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67b123f_64f5_40c1_920c_2bdea84b5b7c.slice/crio-552486d00c0c41f2a7f1007e58f4f0d7a0702f324fd9413a9f8309b93f1a4bd7 WatchSource:0}: Error finding container 552486d00c0c41f2a7f1007e58f4f0d7a0702f324fd9413a9f8309b93f1a4bd7: Status 404 returned error can't find the container with id 552486d00c0c41f2a7f1007e58f4f0d7a0702f324fd9413a9f8309b93f1a4bd7 Sep 19 12:35:08.442150 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.442081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dx6xx\" (UID: \"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.442239 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.442210 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbj6\" (UniqueName: \"kubernetes.io/projected/72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1-kube-api-access-mqbj6\") pod \"payload-pre-processing-59fbc47bf8-dx6xx\" (UID: \"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.542941 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.542908 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbj6\" (UniqueName: \"kubernetes.io/projected/72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1-kube-api-access-mqbj6\") pod \"payload-pre-processing-59fbc47bf8-dx6xx\" (UID: \"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.543118 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.542984 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dx6xx\" (UID: \"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.543846 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.543824 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dx6xx\" (UID: \"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.551187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.551161 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbj6\" (UniqueName: \"kubernetes.io/projected/72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1-kube-api-access-mqbj6\") pod \"payload-pre-processing-59fbc47bf8-dx6xx\" (UID: \"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.698191 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.698107 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:08.904840 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.904804 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" event={"ID":"a67b123f-64f5-40c1-920c-2bdea84b5b7c","Type":"ContainerStarted","Data":"552486d00c0c41f2a7f1007e58f4f0d7a0702f324fd9413a9f8309b93f1a4bd7"} Sep 19 12:35:08.988422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:08.988394 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx"] Sep 19 12:35:08.989393 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:35:08.989351 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fa7a05_6fe2_41f6_acf2_54f1ac7b3bd1.slice/crio-a27b15d2ed2786fbefdd10f88b9fb83fadd5639db27e414278c300a33d1fc584 WatchSource:0}: Error finding container a27b15d2ed2786fbefdd10f88b9fb83fadd5639db27e414278c300a33d1fc584: Status 404 returned error can't find the container with id a27b15d2ed2786fbefdd10f88b9fb83fadd5639db27e414278c300a33d1fc584 Sep 19 12:35:09.913856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:09.913794 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" event={"ID":"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1","Type":"ContainerStarted","Data":"a27b15d2ed2786fbefdd10f88b9fb83fadd5639db27e414278c300a33d1fc584"} Sep 19 12:35:10.922203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:10.922164 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" event={"ID":"a67b123f-64f5-40c1-920c-2bdea84b5b7c","Type":"ContainerStarted","Data":"b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9"} Sep 19 12:35:10.922649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:10.922227 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:10.923829 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:10.923808 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" event={"ID":"72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1","Type":"ContainerStarted","Data":"f3b97bb204c8696585b045fa4844481a2547a9e61766a434264b5f01ede668a0"} Sep 19 12:35:10.923950 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:10.923931 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:10.938200 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:10.938145 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" podStartSLOduration=1.961556037 podStartE2EDuration="3.938130355s" totalImagesPullingTime="1.976574318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:35:10.937795905 +0000 UTC m=+606.544646516" watchObservedRunningTime="2026-09-19 12:35:10.938130355 +0000 UTC m=+606.544980931" Sep 19 12:35:10.953850 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:10.953807 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" podStartSLOduration=1.5223695149999998 podStartE2EDuration="2.953794241s" totalImagesPullingTime="1.431424726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:35:10.952100366 +0000 UTC m=+606.558950949" watchObservedRunningTime="2026-09-19 12:35:10.953794241 +0000 UTC m=+606.560644821" Sep 19 12:35:16.935973 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:16.935945 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 12:35:21.932620 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:21.932585 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dx6xx" Sep 19 12:35:36.129460 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.129426 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz"] Sep 19 12:35:36.143140 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.143103 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz"] Sep 19 12:35:36.143314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.143264 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.146746 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.146724 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pr82q\"" Sep 19 12:35:36.147060 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.146756 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 19 12:35:36.147259 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.146802 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 19 12:35:36.147374 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.146859 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 19 12:35:36.289713 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.289658 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1006afd-75ae-40bd-bde6-04a50948a763-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.289874 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.289764 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.289874 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.289798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.289874 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.289828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.289874 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.289847 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lgt\" (UniqueName: \"kubernetes.io/projected/b1006afd-75ae-40bd-bde6-04a50948a763-kube-api-access-47lgt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.290032 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.289948 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.390983 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.390901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.390983 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.390961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1006afd-75ae-40bd-bde6-04a50948a763-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391205 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.390993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391205 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.391015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391205 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.391040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391205 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.391055 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lgt\" (UniqueName: \"kubernetes.io/projected/b1006afd-75ae-40bd-bde6-04a50948a763-kube-api-access-47lgt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391400 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.391378 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.391444 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.391523 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.391498 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.393335 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.393312 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1006afd-75ae-40bd-bde6-04a50948a763-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.393599 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.393582 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1006afd-75ae-40bd-bde6-04a50948a763-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.398065 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.398043 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lgt\" (UniqueName: \"kubernetes.io/projected/b1006afd-75ae-40bd-bde6-04a50948a763-kube-api-access-47lgt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rjpkz\" (UID: \"b1006afd-75ae-40bd-bde6-04a50948a763\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.454757 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.454734 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:36.731148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.731127 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz"] Sep 19 12:35:36.733073 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:35:36.733047 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1006afd_75ae_40bd_bde6_04a50948a763.slice/crio-3baf9d95c615d83d0b2529aeec66337096b3bce91955165c0098693d5b7cfad0 WatchSource:0}: Error finding container 3baf9d95c615d83d0b2529aeec66337096b3bce91955165c0098693d5b7cfad0: Status 404 returned error can't find the container with id 3baf9d95c615d83d0b2529aeec66337096b3bce91955165c0098693d5b7cfad0 Sep 19 12:35:36.734296 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:36.734277 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 12:35:37.084907 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:37.084864 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" event={"ID":"b1006afd-75ae-40bd-bde6-04a50948a763","Type":"ContainerStarted","Data":"3baf9d95c615d83d0b2529aeec66337096b3bce91955165c0098693d5b7cfad0"} Sep 19 12:35:38.093482 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:38.093451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" event={"ID":"b1006afd-75ae-40bd-bde6-04a50948a763","Type":"ContainerStarted","Data":"575201dac0332dae60c01a281aad59edba2823bee43da7714e6fc2b8c4f04c71"} Sep 19 12:35:38.111349 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:38.111305 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" podStartSLOduration=1.071359976 podStartE2EDuration="2.111289915s" totalImagesPullingTime="1.039929939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:35:38.108615915 +0000 UTC m=+633.715466539" watchObservedRunningTime="2026-09-19 12:35:38.111289915 +0000 UTC m=+633.718140497" Sep 19 12:35:46.455318 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:46.455282 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:46.455796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:46.455346 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:46.468430 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:46.468402 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:35:47.161393 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:35:47.161360 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rjpkz" Sep 19 12:37:42.818490 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:42.818452 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d87c84dd6-nqts2"] Sep 19 12:37:42.819062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:42.818651 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" podUID="867e3b4a-baa2-44d6-ac98-193cb7d2740f" containerName="authorino" containerID="cri-o://da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25" gracePeriod=30 Sep 19 12:37:43.070768 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.070706 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:37:43.131870 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.131836 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/867e3b4a-baa2-44d6-ac98-193cb7d2740f-tls-cert\") pod \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " Sep 19 12:37:43.131870 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.131875 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvpw9\" (UniqueName: \"kubernetes.io/projected/867e3b4a-baa2-44d6-ac98-193cb7d2740f-kube-api-access-kvpw9\") pod \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\" (UID: \"867e3b4a-baa2-44d6-ac98-193cb7d2740f\") " Sep 19 12:37:43.134086 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.134056 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867e3b4a-baa2-44d6-ac98-193cb7d2740f-kube-api-access-kvpw9" (OuterVolumeSpecName: "kube-api-access-kvpw9") pod "867e3b4a-baa2-44d6-ac98-193cb7d2740f" (UID: "867e3b4a-baa2-44d6-ac98-193cb7d2740f"). InnerVolumeSpecName "kube-api-access-kvpw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:37:43.145771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.145740 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867e3b4a-baa2-44d6-ac98-193cb7d2740f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "867e3b4a-baa2-44d6-ac98-193cb7d2740f" (UID: "867e3b4a-baa2-44d6-ac98-193cb7d2740f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:37:43.232654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.232615 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/867e3b4a-baa2-44d6-ac98-193cb7d2740f-tls-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:37:43.232654 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.232648 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvpw9\" (UniqueName: \"kubernetes.io/projected/867e3b4a-baa2-44d6-ac98-193cb7d2740f-kube-api-access-kvpw9\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:37:43.861731 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.861680 2585 generic.go:356] "Generic (PLEG): container finished" podID="867e3b4a-baa2-44d6-ac98-193cb7d2740f" containerID="da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25" exitCode=0 Sep 19 12:37:43.861731 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.861723 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" event={"ID":"867e3b4a-baa2-44d6-ac98-193cb7d2740f","Type":"ContainerDied","Data":"da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25"} Sep 19 12:37:43.861731 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.861737 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" Sep 19 12:37:43.862251 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.861766 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d87c84dd6-nqts2" event={"ID":"867e3b4a-baa2-44d6-ac98-193cb7d2740f","Type":"ContainerDied","Data":"0ec9f30256afcc1f493f56e000f44acd181c9988bf88388638025504a24ed0b5"} Sep 19 12:37:43.862251 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.861785 2585 scope.go:117] "RemoveContainer" containerID="da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25" Sep 19 12:37:43.872169 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.872148 2585 scope.go:117] "RemoveContainer" containerID="da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25" Sep 19 12:37:43.872456 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:37:43.872434 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25\": container with ID starting with da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25 not found: ID does not exist" containerID="da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25" Sep 19 12:37:43.872538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.872460 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25"} err="failed to get container status \"da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25\": rpc error: code = NotFound desc = could not find container \"da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25\": container with ID starting with da5f5e3380434bdde967c8e09d057096106a9bfc2abaea1550b96bf0330eaf25 not found: ID does not exist" Sep 19 12:37:43.889951 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.889927 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d87c84dd6-nqts2"] Sep 19 12:37:43.903912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:43.903889 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d87c84dd6-nqts2"] Sep 19 12:37:44.580258 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:37:44.580224 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867e3b4a-baa2-44d6-ac98-193cb7d2740f" path="/var/lib/kubelet/pods/867e3b4a-baa2-44d6-ac98-193cb7d2740f/volumes" Sep 19 12:38:01.149243 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.149203 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw"] Sep 19 12:38:01.149693 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.149668 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867e3b4a-baa2-44d6-ac98-193cb7d2740f" containerName="authorino" Sep 19 12:38:01.149747 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.149717 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="867e3b4a-baa2-44d6-ac98-193cb7d2740f" containerName="authorino" Sep 19 12:38:01.149819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.149809 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="867e3b4a-baa2-44d6-ac98-193cb7d2740f" containerName="authorino" Sep 19 12:38:01.153002 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.152978 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.155466 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.155444 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 19 12:38:01.155554 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.155446 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a2e924-openshift-default-dockercfg-4mrqx\"" Sep 19 12:38:01.166153 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.166126 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw"] Sep 19 12:38:01.224923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.224892 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v"] Sep 19 12:38:01.233365 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.233329 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.238041 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.237925 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-04a668-openshift-default-dockercfg-9mhv9\"" Sep 19 12:38:01.243051 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.243017 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v"] Sep 19 12:38:01.300051 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-credential-socket\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300063 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-certs\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300130 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-token\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300161 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-podinfo\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300206 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istiod-ca-cert\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300291 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-envoy\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300325 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57b4k\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-kube-api-access-57b4k\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300415 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300360 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-socket\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.300415 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.300396 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-data\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.400921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.400844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-envoy\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.400921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.400884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-data\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.400921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.400902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57b4k\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-kube-api-access-57b4k\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.400923 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-credential-socket\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.400949 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-certs\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.400981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-socket\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-socket\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401074 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-data\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401207 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401152 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-credential-socket\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401293 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6af4c68-7371-4196-a2eb-961dbf2229a5-istiod-ca-cert\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401338 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-certs\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401378 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-envoy\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401447 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-socket\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-token\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401558 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkr5j\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-kube-api-access-kkr5j\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401605 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-podinfo\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401713 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istiod-ca-cert\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401744 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-token\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401938 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401779 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-podinfo\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.401938 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401815 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-certs\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.401938 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401861 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-credential-socket\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.402049 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.401992 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-data\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.402346 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.402328 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istiod-ca-cert\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.403471 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.403447 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-envoy\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.404430 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.404410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-podinfo\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.409178 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.409159 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-token\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.409307 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.409202 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57b4k\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-kube-api-access-57b4k\") pod \"e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.467204 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.467186 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:01.502610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502584 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-token\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502749 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-podinfo\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502749 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502667 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-data\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502749 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502703 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-credential-socket\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502749 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-certs\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502775 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-socket\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6af4c68-7371-4196-a2eb-961dbf2229a5-istiod-ca-cert\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502894 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-envoy\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.502959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.502927 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkr5j\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-kube-api-access-kkr5j\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.503180 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.503129 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-credential-socket\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.503586 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.503508 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-data\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.503586 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.503541 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-certs\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.503770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.503646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6af4c68-7371-4196-a2eb-961dbf2229a5-istiod-ca-cert\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.504055 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.504026 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-socket\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.505601 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.505576 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-podinfo\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.505843 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.505824 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-envoy\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.512074 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.511872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-token\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.512074 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.512033 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkr5j\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-kube-api-access-kkr5j\") pod \"e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.549098 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.549071 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:01.750599 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.750576 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw"] Sep 19 12:38:01.751550 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:01.751523 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd27e7d2_fe3c_4ee5_a3aa_b0866e8e1687.slice/crio-092b9a5b0b443b306b1cf4f7ad5cecd59016d5a0c530fc26b39b2ab36fe4de44 WatchSource:0}: Error finding container 092b9a5b0b443b306b1cf4f7ad5cecd59016d5a0c530fc26b39b2ab36fe4de44: Status 404 returned error can't find the container with id 092b9a5b0b443b306b1cf4f7ad5cecd59016d5a0c530fc26b39b2ab36fe4de44 Sep 19 12:38:01.831019 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.830984 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v"] Sep 19 12:38:01.831893 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:01.831867 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6af4c68_7371_4196_a2eb_961dbf2229a5.slice/crio-6d2b012efcfdecf7123fd247a22db7642c1bf4f3feac27369724a0b3cff5251d WatchSource:0}: Error finding container 6d2b012efcfdecf7123fd247a22db7642c1bf4f3feac27369724a0b3cff5251d: Status 404 returned error can't find the container with id 6d2b012efcfdecf7123fd247a22db7642c1bf4f3feac27369724a0b3cff5251d Sep 19 12:38:01.968379 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.968343 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" event={"ID":"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687","Type":"ContainerStarted","Data":"092b9a5b0b443b306b1cf4f7ad5cecd59016d5a0c530fc26b39b2ab36fe4de44"} Sep 19 12:38:01.969982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:01.969958 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" event={"ID":"d6af4c68-7371-4196-a2eb-961dbf2229a5","Type":"ContainerStarted","Data":"6d2b012efcfdecf7123fd247a22db7642c1bf4f3feac27369724a0b3cff5251d"} Sep 19 12:38:02.685057 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.684548 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7576-d67df5"] Sep 19 12:38:02.688908 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.688879 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:02.691314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.691287 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pjdxd\"" Sep 19 12:38:02.693884 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.693860 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-d67df5"] Sep 19 12:38:02.817433 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.817388 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98tt\" (UniqueName: \"kubernetes.io/projected/9f6251a6-8df0-4165-a295-d1b3aecd2bce-kube-api-access-r98tt\") pod \"test-curl-7576-d67df5\" (UID: \"9f6251a6-8df0-4165-a295-d1b3aecd2bce\") " pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:02.918528 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.918294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98tt\" (UniqueName: \"kubernetes.io/projected/9f6251a6-8df0-4165-a295-d1b3aecd2bce-kube-api-access-r98tt\") pod \"test-curl-7576-d67df5\" (UID: \"9f6251a6-8df0-4165-a295-d1b3aecd2bce\") " pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:02.927336 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:02.927307 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98tt\" (UniqueName: \"kubernetes.io/projected/9f6251a6-8df0-4165-a295-d1b3aecd2bce-kube-api-access-r98tt\") pod \"test-curl-7576-d67df5\" (UID: \"9f6251a6-8df0-4165-a295-d1b3aecd2bce\") " pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:03.005659 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:03.005557 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:03.305282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:03.305254 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-d67df5"] Sep 19 12:38:03.307242 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:03.307208 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f6251a6_8df0_4165_a295_d1b3aecd2bce.slice/crio-221a5a347b629211ea7e5988d2fc13230d20f36142f654436a1a179910a2903b WatchSource:0}: Error finding container 221a5a347b629211ea7e5988d2fc13230d20f36142f654436a1a179910a2903b: Status 404 returned error can't find the container with id 221a5a347b629211ea7e5988d2fc13230d20f36142f654436a1a179910a2903b Sep 19 12:38:03.991323 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:03.991284 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-d67df5" event={"ID":"9f6251a6-8df0-4165-a295-d1b3aecd2bce","Type":"ContainerStarted","Data":"221a5a347b629211ea7e5988d2fc13230d20f36142f654436a1a179910a2903b"} Sep 19 12:38:04.720544 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.720504 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:04.720717 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.720605 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:04.720717 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.720654 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:04.726866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.726840 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:04.726950 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.726897 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:04.726950 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.726925 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:04.988903 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.988811 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9"] Sep 19 12:38:04.994470 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:04.994440 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.001670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.001641 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-61583b-serving-cert\"" Sep 19 12:38:05.026388 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.004933 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr888\"" Sep 19 12:38:05.035887 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.035660 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" event={"ID":"d6af4c68-7371-4196-a2eb-961dbf2229a5","Type":"ContainerStarted","Data":"dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d"} Sep 19 12:38:05.040307 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.040285 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9"] Sep 19 12:38:05.048226 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.048192 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-metrics-certs\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.048343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.048246 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjps\" (UniqueName: \"kubernetes.io/projected/10ab80c1-7e37-4157-8732-ee49da02ef3f-kube-api-access-drjps\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.048343 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.048324 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.048561 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.048528 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab80c1-7e37-4157-8732-ee49da02ef3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.049256 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.049216 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-d67df5" event={"ID":"9f6251a6-8df0-4165-a295-d1b3aecd2bce","Type":"ContainerStarted","Data":"82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35"} Sep 19 12:38:05.055261 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.054987 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" event={"ID":"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687","Type":"ContainerStarted","Data":"b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882"} Sep 19 12:38:05.077348 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.077296 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podStartSLOduration=1.191080332 podStartE2EDuration="4.077274869s" totalImagesPullingTime="2.886194537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:05.058676131 +0000 UTC m=+780.665526731" watchObservedRunningTime="2026-09-19 12:38:05.077274869 +0000 UTC m=+780.684125449" Sep 19 12:38:05.096407 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.096360 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7576-d67df5" podStartSLOduration=1.6861328599999998 podStartE2EDuration="3.096340971s" totalImagesPullingTime="1.410208111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:05.076665646 +0000 UTC m=+780.683516224" watchObservedRunningTime="2026-09-19 12:38:05.096340971 +0000 UTC m=+780.703191552" Sep 19 12:38:05.096982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.096942 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podStartSLOduration=1.1239552210000001 podStartE2EDuration="4.096930176s" totalImagesPullingTime="2.972974955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:05.092773581 +0000 UTC m=+780.699624192" watchObservedRunningTime="2026-09-19 12:38:05.096930176 +0000 UTC m=+780.703780756" Sep 19 12:38:05.149310 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.149273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab80c1-7e37-4157-8732-ee49da02ef3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.149490 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.149368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-metrics-certs\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.149490 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.149444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjps\" (UniqueName: \"kubernetes.io/projected/10ab80c1-7e37-4157-8732-ee49da02ef3f-kube-api-access-drjps\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.149608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.149588 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.149911 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:05.149888 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-serving-cert: secret "maas-api-e2e-inf-a-61583b-serving-cert" not found Sep 19 12:38:05.150046 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:05.149963 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls podName:10ab80c1-7e37-4157-8732-ee49da02ef3f nodeName:}" failed. No retries permitted until 2026-09-19 12:38:05.649944745 +0000 UTC m=+781.256795304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls") pod "maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" (UID: "10ab80c1-7e37-4157-8732-ee49da02ef3f") : secret "maas-api-e2e-inf-a-61583b-serving-cert" not found Sep 19 12:38:05.150353 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.150331 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab80c1-7e37-4157-8732-ee49da02ef3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.152526 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.152503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-metrics-certs\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.158814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.158788 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjps\" (UniqueName: \"kubernetes.io/projected/10ab80c1-7e37-4157-8732-ee49da02ef3f-kube-api-access-drjps\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.467993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.467960 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:05.474720 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.474674 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:05.549418 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.549383 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:05.554243 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.554218 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:05.646077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.646044 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7576-d67df5"] Sep 19 12:38:05.651668 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.651642 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7576-d67df5"] Sep 19 12:38:05.653651 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.653619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.656528 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.656500 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.769955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.769633 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7576-fd9475"] Sep 19 12:38:05.770439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.770414 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" containerName="test-curl-7576-d67df5" Sep 19 12:38:05.770439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.770442 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" containerName="test-curl-7576-d67df5" Sep 19 12:38:05.770603 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.770585 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" containerName="test-curl-7576-d67df5" Sep 19 12:38:05.775121 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.775097 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:05.784730 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.784523 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-fd9475"] Sep 19 12:38:05.855172 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.855142 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvkx\" (UniqueName: \"kubernetes.io/projected/40e65157-c693-4160-a100-8b8fc660dd96-kube-api-access-vxvkx\") pod \"test-curl-7576-fd9475\" (UID: \"40e65157-c693-4160-a100-8b8fc660dd96\") " pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:05.926188 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.926154 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:05.956464 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.956435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvkx\" (UniqueName: \"kubernetes.io/projected/40e65157-c693-4160-a100-8b8fc660dd96-kube-api-access-vxvkx\") pod \"test-curl-7576-fd9475\" (UID: \"40e65157-c693-4160-a100-8b8fc660dd96\") " pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:05.963852 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:05.963829 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvkx\" (UniqueName: \"kubernetes.io/projected/40e65157-c693-4160-a100-8b8fc660dd96-kube-api-access-vxvkx\") pod \"test-curl-7576-fd9475\" (UID: \"40e65157-c693-4160-a100-8b8fc660dd96\") " pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:06.061491 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:06.061455 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7576-d67df5" container="test-curl-7576-d67df5" Sep 19 12:38:06.061913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.061705 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:06.061913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.061752 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:06.062623 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.062601 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:06.062731 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.062663 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:06.087441 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.087406 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:06.205049 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.205014 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9"] Sep 19 12:38:06.206929 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:06.206884 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ab80c1_7e37_4157_8732_ee49da02ef3f.slice/crio-ed0e81ebd3bce8c88403a73dcf4d6542735ef13550279152e473f1bdd3ad1dcf WatchSource:0}: Error finding container ed0e81ebd3bce8c88403a73dcf4d6542735ef13550279152e473f1bdd3ad1dcf: Status 404 returned error can't find the container with id ed0e81ebd3bce8c88403a73dcf4d6542735ef13550279152e473f1bdd3ad1dcf Sep 19 12:38:06.387649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:06.387620 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7576-fd9475"] Sep 19 12:38:06.389259 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:06.389219 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40e65157_c693_4160_a100_8b8fc660dd96.slice/crio-9fc13b0327c6ad0bf4062baba37c34138679cb7d41922db1df45073d1f48c8a1 WatchSource:0}: Error finding container 9fc13b0327c6ad0bf4062baba37c34138679cb7d41922db1df45073d1f48c8a1: Status 404 returned error can't find the container with id 9fc13b0327c6ad0bf4062baba37c34138679cb7d41922db1df45073d1f48c8a1 Sep 19 12:38:07.069995 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.069955 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-fd9475" event={"ID":"40e65157-c693-4160-a100-8b8fc660dd96","Type":"ContainerStarted","Data":"9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728"} Sep 19 12:38:07.070461 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.070029 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7576-fd9475" event={"ID":"40e65157-c693-4160-a100-8b8fc660dd96","Type":"ContainerStarted","Data":"9fc13b0327c6ad0bf4062baba37c34138679cb7d41922db1df45073d1f48c8a1"} Sep 19 12:38:07.072058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.072024 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" event={"ID":"10ab80c1-7e37-4157-8732-ee49da02ef3f","Type":"ContainerStarted","Data":"a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc"} Sep 19 12:38:07.072221 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.072093 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" event={"ID":"10ab80c1-7e37-4157-8732-ee49da02ef3f","Type":"ContainerStarted","Data":"ed0e81ebd3bce8c88403a73dcf4d6542735ef13550279152e473f1bdd3ad1dcf"} Sep 19 12:38:07.072337 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.072299 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7576-d67df5" podUID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" containerName="test-curl-7576-d67df5" containerID="cri-o://82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35" gracePeriod=2 Sep 19 12:38:07.072402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.072360 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:07.084866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.084822 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7576-fd9475" podStartSLOduration=2.084808474 podStartE2EDuration="2.084808474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:07.083436222 +0000 UTC m=+782.690286800" watchObservedRunningTime="2026-09-19 12:38:07.084808474 +0000 UTC m=+782.691659453" Sep 19 12:38:07.101359 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.101313 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" podStartSLOduration=2.687381325 podStartE2EDuration="3.101298761s" totalImagesPullingTime="413.917436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:07.100725957 +0000 UTC m=+782.707576563" watchObservedRunningTime="2026-09-19 12:38:07.101298761 +0000 UTC m=+782.708149340" Sep 19 12:38:07.722807 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.722770 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7576-fd9475"] Sep 19 12:38:07.728427 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:07.728352 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7576-fd9475"] Sep 19 12:38:08.080341 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:08.080305 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7576-fd9475" container="test-curl-7576-fd9475" Sep 19 12:38:08.495196 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.495105 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz"] Sep 19 12:38:08.496027 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.495999 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e65157-c693-4160-a100-8b8fc660dd96" containerName="test-curl-7576-fd9475" Sep 19 12:38:08.496027 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.496026 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e65157-c693-4160-a100-8b8fc660dd96" containerName="test-curl-7576-fd9475" Sep 19 12:38:08.496258 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.496161 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e65157-c693-4160-a100-8b8fc660dd96" containerName="test-curl-7576-fd9475" Sep 19 12:38:08.499829 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.499802 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.502541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.502517 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-38aa0a-openshift-default-dockercfg-s68d2\"" Sep 19 12:38:08.510033 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.510007 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz"] Sep 19 12:38:08.580889 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.580862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14111d82-d34e-4742-9994-df76b8b18400-istio-podinfo\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.580903 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-certs\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.580926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-credential-socket\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.580968 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-socket\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581272 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.581241 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14111d82-d34e-4742-9994-df76b8b18400-istiod-ca-cert\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.581302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-data\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.581366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-envoy\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.581400 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-istio-token\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.581508 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.581431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7skb\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-kube-api-access-d7skb\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.631849 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.631811 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k"] Sep 19 12:38:08.635965 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.635943 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.638508 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.638488 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a2e924-serving-cert\"" Sep 19 12:38:08.647392 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.647368 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k"] Sep 19 12:38:08.682317 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-envoy\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.682468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-istio-token\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.682468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682353 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7skb\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-kube-api-access-d7skb\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.682468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682429 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5720c5e4-7bcf-4881-9a62-28e38d60e042-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.682649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682480 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhr46\" (UniqueName: \"kubernetes.io/projected/5720c5e4-7bcf-4881-9a62-28e38d60e042-kube-api-access-fhr46\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.682649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.682649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682557 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-metrics-certs\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.682649 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14111d82-d34e-4742-9994-df76b8b18400-istio-podinfo\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.682908 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-certs\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.682908 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.682898 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-credential-socket\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-socket\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14111d82-d34e-4742-9994-df76b8b18400-istiod-ca-cert\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683182 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683160 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-data\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683235 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683179 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-certs\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683459 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-credential-socket\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-data\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683750 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683719 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14111d82-d34e-4742-9994-df76b8b18400-istiod-ca-cert\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.683845 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.683829 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-socket\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.685725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.685671 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-envoy\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.686025 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.686005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14111d82-d34e-4742-9994-df76b8b18400-istio-podinfo\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.690391 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.690371 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-istio-token\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.690982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.690960 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7skb\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-kube-api-access-d7skb\") pod \"e2e-autores-38aa0a-openshift-default-656d955f98-m45jz\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:08.784576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.784489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5720c5e4-7bcf-4881-9a62-28e38d60e042-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.784576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.784530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhr46\" (UniqueName: \"kubernetes.io/projected/5720c5e4-7bcf-4881-9a62-28e38d60e042-kube-api-access-fhr46\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.784576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.784557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.784576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.784574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-metrics-certs\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.784885 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:08.784747 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-serving-cert: secret "maas-api-e2e-worker-w0-a2e924-serving-cert" not found Sep 19 12:38:08.784885 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:08.784838 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls podName:5720c5e4-7bcf-4881-9a62-28e38d60e042 nodeName:}" failed. No retries permitted until 2026-09-19 12:38:09.284814119 +0000 UTC m=+784.891664680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls") pod "maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" (UID: "5720c5e4-7bcf-4881-9a62-28e38d60e042") : secret "maas-api-e2e-worker-w0-a2e924-serving-cert" not found Sep 19 12:38:08.785278 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.785254 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5720c5e4-7bcf-4881-9a62-28e38d60e042-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.787281 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.787260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-metrics-certs\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.796317 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.796295 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhr46\" (UniqueName: \"kubernetes.io/projected/5720c5e4-7bcf-4881-9a62-28e38d60e042-kube-api-access-fhr46\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:08.815223 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:08.815200 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:09.087351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.087315 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7576-fd9475" podUID="40e65157-c693-4160-a100-8b8fc660dd96" containerName="test-curl-7576-fd9475" containerID="cri-o://9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728" gracePeriod=2 Sep 19 12:38:09.096930 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.096906 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz"] Sep 19 12:38:09.117925 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:09.117896 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14111d82_d34e_4742_9994_df76b8b18400.slice/crio-bd8fc4c2cd67500a116a96ddd9bd65886ac356ee08cbeca7700d07de8f50d03c WatchSource:0}: Error finding container bd8fc4c2cd67500a116a96ddd9bd65886ac356ee08cbeca7700d07de8f50d03c: Status 404 returned error can't find the container with id bd8fc4c2cd67500a116a96ddd9bd65886ac356ee08cbeca7700d07de8f50d03c Sep 19 12:38:09.120062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.120032 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:09.120129 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.120096 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:09.120181 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.120136 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:09.253792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.253759 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:09.290559 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.290458 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r98tt\" (UniqueName: \"kubernetes.io/projected/9f6251a6-8df0-4165-a295-d1b3aecd2bce-kube-api-access-r98tt\") pod \"9f6251a6-8df0-4165-a295-d1b3aecd2bce\" (UID: \"9f6251a6-8df0-4165-a295-d1b3aecd2bce\") " Sep 19 12:38:09.290727 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.290660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:09.294000 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.293961 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6251a6-8df0-4165-a295-d1b3aecd2bce-kube-api-access-r98tt" (OuterVolumeSpecName: "kube-api-access-r98tt") pod "9f6251a6-8df0-4165-a295-d1b3aecd2bce" (UID: "9f6251a6-8df0-4165-a295-d1b3aecd2bce"). InnerVolumeSpecName "kube-api-access-r98tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:38:09.294000 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.293991 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:09.392018 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.391946 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r98tt\" (UniqueName: \"kubernetes.io/projected/9f6251a6-8df0-4165-a295-d1b3aecd2bce-kube-api-access-r98tt\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:38:09.550863 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.550831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:09.837814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:09.837785 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k"] Sep 19 12:38:10.095048 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.095017 2585 generic.go:356] "Generic (PLEG): container finished" podID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" containerID="82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35" exitCode=137 Sep 19 12:38:10.095522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.095095 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:10.095522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.095130 2585 scope.go:117] "RemoveContainer" containerID="82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35" Sep 19 12:38:10.096541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.096519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" event={"ID":"5720c5e4-7bcf-4881-9a62-28e38d60e042","Type":"ContainerStarted","Data":"caf57563481f15f7d2feb9dd8fb62de7d603e395c9d3d8f59d8091833596f882"} Sep 19 12:38:10.098465 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.098241 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" event={"ID":"14111d82-d34e-4742-9994-df76b8b18400","Type":"ContainerStarted","Data":"6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c"} Sep 19 12:38:10.098465 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.098305 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" event={"ID":"14111d82-d34e-4742-9994-df76b8b18400","Type":"ContainerStarted","Data":"bd8fc4c2cd67500a116a96ddd9bd65886ac356ee08cbeca7700d07de8f50d03c"} Sep 19 12:38:10.105910 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.105893 2585 scope.go:117] "RemoveContainer" containerID="82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35" Sep 19 12:38:10.106231 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:10.106205 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35\": container with ID starting with 82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35 not found: ID does not exist" containerID="82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35" Sep 19 12:38:10.106282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.106243 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35"} err="failed to get container status \"82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35\": rpc error: code = NotFound desc = could not find container \"82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35\": container with ID starting with 82d29394f3d2a8710e9fab169c38a5e625d71200b9abc5e460d5fbcc85826f35 not found: ID does not exist" Sep 19 12:38:10.118420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.118370 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podStartSLOduration=2.118357299 podStartE2EDuration="2.118357299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:10.116877628 +0000 UTC m=+785.723728228" watchObservedRunningTime="2026-09-19 12:38:10.118357299 +0000 UTC m=+785.725207878" Sep 19 12:38:10.118619 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:10.118595 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7576-d67df5\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" pod="opendatahub/test-curl-7576-d67df5" Sep 19 12:38:10.580058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.580022 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6251a6-8df0-4165-a295-d1b3aecd2bce" path="/var/lib/kubelet/pods/9f6251a6-8df0-4165-a295-d1b3aecd2bce/volumes" Sep 19 12:38:10.601840 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.601811 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn"] Sep 19 12:38:10.605924 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.605901 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.608586 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.608561 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-04a668\"" Sep 19 12:38:10.609061 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.609033 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-04a668-dockercfg-vsgwm\"" Sep 19 12:38:10.609726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.609705 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9"] Sep 19 12:38:10.613506 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.613483 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.615765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.615746 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn"] Sep 19 12:38:10.624218 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.624192 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9"] Sep 19 12:38:10.706643 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.706595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f698e-3440-4f04-be22-841180eefc02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.706847 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.706655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gwf\" (UniqueName: \"kubernetes.io/projected/489f698e-3440-4f04-be22-841180eefc02-kube-api-access-z8gwf\") pod \"payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.706847 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.706824 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfrl\" (UniqueName: \"kubernetes.io/projected/4ff3827c-b37f-4acd-b2bb-a886730f2897-kube-api-access-xvfrl\") pod \"payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.706979 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.706852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff3827c-b37f-4acd-b2bb-a886730f2897-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.807828 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.807792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f698e-3440-4f04-be22-841180eefc02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.808016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.807837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gwf\" (UniqueName: \"kubernetes.io/projected/489f698e-3440-4f04-be22-841180eefc02-kube-api-access-z8gwf\") pod \"payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.808016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.807910 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfrl\" (UniqueName: \"kubernetes.io/projected/4ff3827c-b37f-4acd-b2bb-a886730f2897-kube-api-access-xvfrl\") pod \"payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.808016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.807937 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff3827c-b37f-4acd-b2bb-a886730f2897-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.808522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.808503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f698e-3440-4f04-be22-841180eefc02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.808728 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.808670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff3827c-b37f-4acd-b2bb-a886730f2897-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.815389 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.815368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gwf\" (UniqueName: \"kubernetes.io/projected/489f698e-3440-4f04-be22-841180eefc02-kube-api-access-z8gwf\") pod \"payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:10.815517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.815495 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfrl\" (UniqueName: \"kubernetes.io/projected/4ff3827c-b37f-4acd-b2bb-a886730f2897-kube-api-access-xvfrl\") pod \"payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.815563 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.815514 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:10.821819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.821796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:10.920506 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.920415 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:10.929564 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:10.929521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:11.117662 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.117623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" event={"ID":"5720c5e4-7bcf-4881-9a62-28e38d60e042","Type":"ContainerStarted","Data":"2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9"} Sep 19 12:38:11.118299 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.118238 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:11.118299 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.118276 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:11.119528 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.119504 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:11.135276 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.135220 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" podStartSLOduration=2.7965046620000003 podStartE2EDuration="3.135200845s" totalImagesPullingTime="338.696183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:11.134486578 +0000 UTC m=+786.741337185" watchObservedRunningTime="2026-09-19 12:38:11.135200845 +0000 UTC m=+786.742051426" Sep 19 12:38:11.216903 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.216878 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn"] Sep 19 12:38:11.218572 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:11.218546 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff3827c_b37f_4acd_b2bb_a886730f2897.slice/crio-e4e4640922d724fe272b280032adaf9b40441a08b5fab7874754402e7b521756 WatchSource:0}: Error finding container e4e4640922d724fe272b280032adaf9b40441a08b5fab7874754402e7b521756: Status 404 returned error can't find the container with id e4e4640922d724fe272b280032adaf9b40441a08b5fab7874754402e7b521756 Sep 19 12:38:11.225139 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:11.225103 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489f698e_3440_4f04_be22_841180eefc02.slice/crio-39fe7272cd82a38596f47bfc1c95661d55868add71ab87c304e075f1feaba2e6 WatchSource:0}: Error finding container 39fe7272cd82a38596f47bfc1c95661d55868add71ab87c304e075f1feaba2e6: Status 404 returned error can't find the container with id 39fe7272cd82a38596f47bfc1c95661d55868add71ab87c304e075f1feaba2e6 Sep 19 12:38:11.225540 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.225449 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9"] Sep 19 12:38:11.282776 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.282750 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:11.313350 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.313327 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxvkx\" (UniqueName: \"kubernetes.io/projected/40e65157-c693-4160-a100-8b8fc660dd96-kube-api-access-vxvkx\") pod \"40e65157-c693-4160-a100-8b8fc660dd96\" (UID: \"40e65157-c693-4160-a100-8b8fc660dd96\") " Sep 19 12:38:11.315596 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.315568 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e65157-c693-4160-a100-8b8fc660dd96-kube-api-access-vxvkx" (OuterVolumeSpecName: "kube-api-access-vxvkx") pod "40e65157-c693-4160-a100-8b8fc660dd96" (UID: "40e65157-c693-4160-a100-8b8fc660dd96"). InnerVolumeSpecName "kube-api-access-vxvkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:38:11.414453 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:11.414418 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvkx\" (UniqueName: \"kubernetes.io/projected/40e65157-c693-4160-a100-8b8fc660dd96-kube-api-access-vxvkx\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:38:12.127931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.127840 2585 generic.go:356] "Generic (PLEG): container finished" podID="40e65157-c693-4160-a100-8b8fc660dd96" containerID="9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728" exitCode=137 Sep 19 12:38:12.127931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.127893 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:12.128424 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.127923 2585 scope.go:117] "RemoveContainer" containerID="9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728" Sep 19 12:38:12.130021 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.129995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" event={"ID":"489f698e-3440-4f04-be22-841180eefc02","Type":"ContainerStarted","Data":"9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6"} Sep 19 12:38:12.130171 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.130040 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" event={"ID":"489f698e-3440-4f04-be22-841180eefc02","Type":"ContainerStarted","Data":"39fe7272cd82a38596f47bfc1c95661d55868add71ab87c304e075f1feaba2e6"} Sep 19 12:38:12.130725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.130705 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:12.132830 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.132461 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" event={"ID":"4ff3827c-b37f-4acd-b2bb-a886730f2897","Type":"ContainerStarted","Data":"388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955"} Sep 19 12:38:12.132830 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.132509 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" event={"ID":"4ff3827c-b37f-4acd-b2bb-a886730f2897","Type":"ContainerStarted","Data":"e4e4640922d724fe272b280032adaf9b40441a08b5fab7874754402e7b521756"} Sep 19 12:38:12.132830 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.132628 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:12.138439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.138421 2585 scope.go:117] "RemoveContainer" containerID="9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728" Sep 19 12:38:12.138736 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:12.138716 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728\": container with ID starting with 9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728 not found: ID does not exist" containerID="9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728" Sep 19 12:38:12.138796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.138745 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728"} err="failed to get container status \"9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728\": rpc error: code = NotFound desc = could not find container \"9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728\": container with ID starting with 9d7e74de5f7024fda576bcf6515254d1e35b924722702d1ebe0e302887abe728 not found: ID does not exist" Sep 19 12:38:12.167924 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:12.167881 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7576-fd9475\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="40e65157-c693-4160-a100-8b8fc660dd96" pod="opendatahub/test-curl-7576-fd9475" Sep 19 12:38:12.168510 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.168458 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" podStartSLOduration=2.16844292 podStartE2EDuration="2.16844292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:12.165495025 +0000 UTC m=+787.772345625" watchObservedRunningTime="2026-09-19 12:38:12.16844292 +0000 UTC m=+787.775293500" Sep 19 12:38:12.202551 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.202495 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" podStartSLOduration=2.202476165 podStartE2EDuration="2.202476165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:12.200482045 +0000 UTC m=+787.807332649" watchObservedRunningTime="2026-09-19 12:38:12.202476165 +0000 UTC m=+787.809326747" Sep 19 12:38:12.581886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:12.581854 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e65157-c693-4160-a100-8b8fc660dd96" path="/var/lib/kubelet/pods/40e65157-c693-4160-a100-8b8fc660dd96/volumes" Sep 19 12:38:14.100928 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:14.100895 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:38:17.648538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.648454 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss"] Sep 19 12:38:17.654579 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.654560 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.657103 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.657081 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-38aa0a-serving-cert\"" Sep 19 12:38:17.664754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.664664 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss"] Sep 19 12:38:17.787707 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.787268 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b281f9e1-e55f-4ddf-b492-775d210281cb-openshift-service-ca\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.787707 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.787366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7s8l\" (UniqueName: \"kubernetes.io/projected/b281f9e1-e55f-4ddf-b492-775d210281cb-kube-api-access-m7s8l\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.787707 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.787397 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.787707 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.787453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-metrics-certs\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.888399 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.888360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-metrics-certs\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.888767 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.888744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b281f9e1-e55f-4ddf-b492-775d210281cb-openshift-service-ca\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.888959 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.888944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7s8l\" (UniqueName: \"kubernetes.io/projected/b281f9e1-e55f-4ddf-b492-775d210281cb-kube-api-access-m7s8l\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.889073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.889060 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.889288 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:17.889272 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-serving-cert: secret "maas-api-e2e-autores-38aa0a-serving-cert" not found Sep 19 12:38:17.889450 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:38:17.889438 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls podName:b281f9e1-e55f-4ddf-b492-775d210281cb nodeName:}" failed. No retries permitted until 2026-09-19 12:38:18.389417314 +0000 UTC m=+793.996267884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls") pod "maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" (UID: "b281f9e1-e55f-4ddf-b492-775d210281cb") : secret "maas-api-e2e-autores-38aa0a-serving-cert" not found Sep 19 12:38:17.889557 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.889517 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b281f9e1-e55f-4ddf-b492-775d210281cb-openshift-service-ca\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.892898 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.892872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-metrics-certs\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:17.900820 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:17.900763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7s8l\" (UniqueName: \"kubernetes.io/projected/b281f9e1-e55f-4ddf-b492-775d210281cb-kube-api-access-m7s8l\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:18.152386 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:18.152300 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:38:18.402640 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:18.402551 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:18.414028 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:18.413995 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls\") pod \"maas-api-e2e-autores-38aa0a-54b7ff9db-qprss\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:18.580202 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:18.579916 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:18.941159 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:18.941111 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss"] Sep 19 12:38:18.942457 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:18.942426 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb281f9e1_e55f_4ddf_b492_775d210281cb.slice/crio-baf48140662ca69da8155b381a44a0f55fdef08f813ba5230c5655f13650b757 WatchSource:0}: Error finding container baf48140662ca69da8155b381a44a0f55fdef08f813ba5230c5655f13650b757: Status 404 returned error can't find the container with id baf48140662ca69da8155b381a44a0f55fdef08f813ba5230c5655f13650b757 Sep 19 12:38:19.110073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.110043 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt"] Sep 19 12:38:19.114209 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.114183 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.116787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.116764 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5be349-openshift-default-dockercfg-6m5gt\"" Sep 19 12:38:19.145147 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.145123 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt"] Sep 19 12:38:19.195213 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.195112 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" event={"ID":"b281f9e1-e55f-4ddf-b492-775d210281cb","Type":"ContainerStarted","Data":"baf48140662ca69da8155b381a44a0f55fdef08f813ba5230c5655f13650b757"} Sep 19 12:38:19.220940 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.220902 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-certs\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.220958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4bz\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-kube-api-access-6t4bz\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-credential-socket\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221234 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221166 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-data\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221234 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-podinfo\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221330 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221255 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-envoy\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221330 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221285 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-socket\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221452 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221429 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-token\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.221512 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.221463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istiod-ca-cert\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-credential-socket\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-data\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-podinfo\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-envoy\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322769 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322588 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-socket\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322769 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322650 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-token\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322769 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istiod-ca-cert\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322917 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322788 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-certs\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.322917 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322820 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4bz\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-kube-api-access-6t4bz\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.323021 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322917 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-credential-socket\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.323021 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.322935 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-data\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.323360 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.323301 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-socket\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.323630 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.323606 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-certs\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.323799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.323776 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istiod-ca-cert\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.325302 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.325274 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-envoy\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.325401 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.325282 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-podinfo\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.332268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.332246 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-token\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.332814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.332793 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4bz\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-kube-api-access-6t4bz\") pod \"e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.428893 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.428863 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:19.713271 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.713235 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt"] Sep 19 12:38:19.713399 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:38:19.713333 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff40a8f_0654_4c20_afd6_95a27fb740e7.slice/crio-a461b7224fca4b67e049b8f033114ca6b1c679a5649709cc1d2b2740baefd2d8 WatchSource:0}: Error finding container a461b7224fca4b67e049b8f033114ca6b1c679a5649709cc1d2b2740baefd2d8: Status 404 returned error can't find the container with id a461b7224fca4b67e049b8f033114ca6b1c679a5649709cc1d2b2740baefd2d8 Sep 19 12:38:19.715799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.715769 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:19.715895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.715839 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:19.715895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:19.715868 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:20.206667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.206627 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" event={"ID":"b281f9e1-e55f-4ddf-b492-775d210281cb","Type":"ContainerStarted","Data":"2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f"} Sep 19 12:38:20.207282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.206713 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:20.208738 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.208708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" event={"ID":"1ff40a8f-0654-4c20-afd6-95a27fb740e7","Type":"ContainerStarted","Data":"f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973"} Sep 19 12:38:20.208841 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.208762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" event={"ID":"1ff40a8f-0654-4c20-afd6-95a27fb740e7","Type":"ContainerStarted","Data":"a461b7224fca4b67e049b8f033114ca6b1c679a5649709cc1d2b2740baefd2d8"} Sep 19 12:38:20.223420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.223379 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" podStartSLOduration=2.898015125 podStartE2EDuration="3.223366164s" totalImagesPullingTime="325.351039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:20.221776574 +0000 UTC m=+795.828627173" watchObservedRunningTime="2026-09-19 12:38:20.223366164 +0000 UTC m=+795.830216743" Sep 19 12:38:20.239126 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.239073 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podStartSLOduration=1.239054835 podStartE2EDuration="1.239054835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:38:20.23866336 +0000 UTC m=+795.845513967" watchObservedRunningTime="2026-09-19 12:38:20.239054835 +0000 UTC m=+795.845905415" Sep 19 12:38:20.429261 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.429224 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:20.435172 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:20.435138 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:21.215385 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:21.215340 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:21.216435 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:21.216413 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:22.101854 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:22.101815 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 19 12:38:22.102026 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:22.101871 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 19 12:38:23.148754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.148650 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:38:23.229976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.229946 2585 generic.go:356] "Generic (PLEG): container finished" podID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerID="b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882" exitCode=0 Sep 19 12:38:23.230166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.230019 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" event={"ID":"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687","Type":"ContainerDied","Data":"b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882"} Sep 19 12:38:23.230536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.230518 2585 scope.go:117] "RemoveContainer" containerID="b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882" Sep 19 12:38:23.231636 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.231610 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:23.231712 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.231672 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:23.231756 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:23.231729 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:24.155315 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:24.155283 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:38:24.243966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:24.243932 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" event={"ID":"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687","Type":"ContainerStarted","Data":"25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d"} Sep 19 12:38:24.467921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:24.467838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:24.468077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:24.468043 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 19 12:38:24.468126 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:24.468095 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 19 12:38:25.468148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:25.468081 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 19 12:38:25.468148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:25.468144 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 19 12:38:26.221020 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:26.220986 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:38:26.468625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:26.468586 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 19 12:38:26.469140 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:26.468641 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 19 12:38:27.468551 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:27.468500 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 19 12:38:27.468851 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:27.468573 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 19 12:38:28.468311 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:28.468272 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 19 12:38:28.468588 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:28.468558 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 19 12:38:29.468085 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:29.468047 2585 patch_prober.go:28] interesting pod/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 19 12:38:29.468597 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:29.468114 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 19 12:38:30.522570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:30.522529 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:30.523846 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:30.523816 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:30.526930 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:30.526908 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:38:36.358972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:36.358456 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:34642->10.132.0.63:15021: read: connection reset by peer" start-of-body= Sep 19 12:38:36.358972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:36.358630 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:34642->10.132.0.63:15021: read: connection reset by peer" Sep 19 12:38:37.360133 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.360097 2585 generic.go:356] "Generic (PLEG): container finished" podID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerID="dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d" exitCode=0 Sep 19 12:38:37.360617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.360167 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" event={"ID":"d6af4c68-7371-4196-a2eb-961dbf2229a5","Type":"ContainerDied","Data":"dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d"} Sep 19 12:38:37.360729 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.360628 2585 scope.go:117] "RemoveContainer" containerID="dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d" Sep 19 12:38:37.361941 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.361901 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:37.362038 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.361994 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:37.362077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.362032 2585 generic.go:356] "Generic (PLEG): container finished" podID="14111d82-d34e-4742-9994-df76b8b18400" containerID="6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c" exitCode=0 Sep 19 12:38:37.362077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.362045 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:37.362144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.362105 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" event={"ID":"14111d82-d34e-4742-9994-df76b8b18400","Type":"ContainerDied","Data":"6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c"} Sep 19 12:38:37.362397 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.362383 2585 scope.go:117] "RemoveContainer" containerID="6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c" Sep 19 12:38:37.363414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.363387 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:37.363504 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.363459 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:37.363566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:37.363510 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:38.371698 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.371650 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" event={"ID":"14111d82-d34e-4742-9994-df76b8b18400","Type":"ContainerStarted","Data":"70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9"} Sep 19 12:38:38.374552 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.374524 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" event={"ID":"d6af4c68-7371-4196-a2eb-961dbf2229a5","Type":"ContainerStarted","Data":"d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3"} Sep 19 12:38:38.549729 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.549700 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:38.549894 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.549796 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 12:38:38.549894 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.549831 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 12:38:38.817051 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.817022 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:38.818160 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.818128 2585 patch_prober.go:28] interesting pod/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 12:38:38.818405 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:38.818376 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 12:38:39.550306 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:39.550275 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 12:38:39.550676 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:39.550330 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 12:38:39.816566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:39.816461 2585 patch_prober.go:28] interesting pod/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 12:38:39.816566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:39.816535 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 12:38:40.550335 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:40.550300 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 12:38:40.550739 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:40.550381 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 12:38:40.816324 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:40.816226 2585 patch_prober.go:28] interesting pod/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 12:38:40.817066 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:40.816596 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 12:38:41.118954 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:41.118859 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:41.549831 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:41.549791 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 12:38:41.550171 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:41.550098 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 12:38:41.816543 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:41.816038 2585 patch_prober.go:28] interesting pod/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 12:38:41.816543 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:41.816104 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 12:38:42.550884 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:42.550847 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 12:38:42.551162 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:42.551131 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 12:38:42.815790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:42.815671 2585 patch_prober.go:28] interesting pod/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 12:38:42.816066 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:42.816036 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 12:38:43.549856 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:43.549813 2585 patch_prober.go:28] interesting pod/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 12:38:43.550339 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:43.549882 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 12:38:43.816797 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:43.816665 2585 patch_prober.go:28] interesting pod/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 12:38:43.817079 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:43.817035 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 12:38:44.555096 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:44.555064 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:44.555847 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:44.555825 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:44.557128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:44.557108 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:38:44.820635 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:44.820563 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:44.821677 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:44.821655 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:38:50.490598 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:50.490564 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerID="f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973" exitCode=0 Sep 19 12:38:50.491035 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:50.490625 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" event={"ID":"1ff40a8f-0654-4c20-afd6-95a27fb740e7","Type":"ContainerDied","Data":"f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973"} Sep 19 12:38:50.491182 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:50.491161 2585 scope.go:117] "RemoveContainer" containerID="f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973" Sep 19 12:38:50.492435 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:50.492398 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:50.492525 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:50.492469 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:50.492525 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:50.492503 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 19 12:38:51.215743 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:51.215710 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:51.502432 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:51.502342 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" event={"ID":"1ff40a8f-0654-4c20-afd6-95a27fb740e7","Type":"ContainerStarted","Data":"3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906"} Sep 19 12:38:52.429472 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:52.429430 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:52.430848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:52.430817 2585 patch_prober.go:28] interesting pod/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 19 12:38:52.430982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:52.430878 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 19 12:38:53.429260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:53.429222 2585 patch_prober.go:28] interesting pod/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 19 12:38:53.429780 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:53.429284 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 19 12:38:54.429182 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:54.429135 2585 patch_prober.go:28] interesting pod/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 19 12:38:54.429383 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:54.429203 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 19 12:38:55.429753 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:55.429711 2585 patch_prober.go:28] interesting pod/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 19 12:38:55.430302 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:55.429812 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 19 12:38:56.429446 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:56.429400 2585 patch_prober.go:28] interesting pod/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 19 12:38:56.429656 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:56.429492 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 19 12:38:57.434144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:57.434117 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:57.434554 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:57.434539 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:38:57.435334 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:38:57.435316 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:40:06.539770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:06.539697 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:40:06.541493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:06.541461 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:40:22.461142 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.461108 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss"] Sep 19 12:40:22.461799 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.461456 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" podUID="b281f9e1-e55f-4ddf-b492-775d210281cb" containerName="maas-api" containerID="cri-o://2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f" gracePeriod=30 Sep 19 12:40:22.775960 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.775915 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:40:22.827184 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.827155 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b281f9e1-e55f-4ddf-b492-775d210281cb-openshift-service-ca\") pod \"b281f9e1-e55f-4ddf-b492-775d210281cb\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " Sep 19 12:40:22.827357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.827200 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7s8l\" (UniqueName: \"kubernetes.io/projected/b281f9e1-e55f-4ddf-b492-775d210281cb-kube-api-access-m7s8l\") pod \"b281f9e1-e55f-4ddf-b492-775d210281cb\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " Sep 19 12:40:22.827357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.827305 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-metrics-certs\") pod \"b281f9e1-e55f-4ddf-b492-775d210281cb\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " Sep 19 12:40:22.827482 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.827368 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls\") pod \"b281f9e1-e55f-4ddf-b492-775d210281cb\" (UID: \"b281f9e1-e55f-4ddf-b492-775d210281cb\") " Sep 19 12:40:22.827568 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.827541 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b281f9e1-e55f-4ddf-b492-775d210281cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b281f9e1-e55f-4ddf-b492-775d210281cb" (UID: "b281f9e1-e55f-4ddf-b492-775d210281cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:40:22.827770 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.827745 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b281f9e1-e55f-4ddf-b492-775d210281cb-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:40:22.829784 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.829752 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b281f9e1-e55f-4ddf-b492-775d210281cb" (UID: "b281f9e1-e55f-4ddf-b492-775d210281cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:40:22.829898 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.829799 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b281f9e1-e55f-4ddf-b492-775d210281cb-kube-api-access-m7s8l" (OuterVolumeSpecName: "kube-api-access-m7s8l") pod "b281f9e1-e55f-4ddf-b492-775d210281cb" (UID: "b281f9e1-e55f-4ddf-b492-775d210281cb"). InnerVolumeSpecName "kube-api-access-m7s8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:40:22.829898 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.829808 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b281f9e1-e55f-4ddf-b492-775d210281cb" (UID: "b281f9e1-e55f-4ddf-b492-775d210281cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:40:22.928951 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.928923 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:40:22.928951 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.928951 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b281f9e1-e55f-4ddf-b492-775d210281cb-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:40:22.929130 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:22.928965 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7s8l\" (UniqueName: \"kubernetes.io/projected/b281f9e1-e55f-4ddf-b492-775d210281cb-kube-api-access-m7s8l\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:40:23.125078 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.125038 2585 generic.go:356] "Generic (PLEG): container finished" podID="b281f9e1-e55f-4ddf-b492-775d210281cb" containerID="2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f" exitCode=0 Sep 19 12:40:23.125252 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.125086 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" Sep 19 12:40:23.125252 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.125082 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" event={"ID":"b281f9e1-e55f-4ddf-b492-775d210281cb","Type":"ContainerDied","Data":"2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f"} Sep 19 12:40:23.125252 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.125208 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss" event={"ID":"b281f9e1-e55f-4ddf-b492-775d210281cb","Type":"ContainerDied","Data":"baf48140662ca69da8155b381a44a0f55fdef08f813ba5230c5655f13650b757"} Sep 19 12:40:23.125252 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.125231 2585 scope.go:117] "RemoveContainer" containerID="2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f" Sep 19 12:40:23.135531 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.135515 2585 scope.go:117] "RemoveContainer" containerID="2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f" Sep 19 12:40:23.135813 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:40:23.135796 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f\": container with ID starting with 2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f not found: ID does not exist" containerID="2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f" Sep 19 12:40:23.135880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.135820 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f"} err="failed to get container status \"2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f\": rpc error: code = NotFound desc = could not find container \"2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f\": container with ID starting with 2df5aabffd5ccda2de58be6fc9dd8551b4bab835a314dcbd648581bc97ff1b2f not found: ID does not exist" Sep 19 12:40:23.148121 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.148100 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss"] Sep 19 12:40:23.150973 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:23.150953 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38aa0a-54b7ff9db-qprss"] Sep 19 12:40:24.580746 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:24.580717 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b281f9e1-e55f-4ddf-b492-775d210281cb" path="/var/lib/kubelet/pods/b281f9e1-e55f-4ddf-b492-775d210281cb/volumes" Sep 19 12:40:26.746482 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.746449 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj"] Sep 19 12:40:26.747283 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.747262 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b281f9e1-e55f-4ddf-b492-775d210281cb" containerName="maas-api" Sep 19 12:40:26.747283 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.747284 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b281f9e1-e55f-4ddf-b492-775d210281cb" containerName="maas-api" Sep 19 12:40:26.747439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.747414 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b281f9e1-e55f-4ddf-b492-775d210281cb" containerName="maas-api" Sep 19 12:40:26.751129 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.751105 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.753543 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.753513 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-21b2975e-serving-cert\"" Sep 19 12:40:26.763942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.763910 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj"] Sep 19 12:40:26.868700 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.868644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjjh\" (UniqueName: \"kubernetes.io/projected/ef3e9ab6-e956-4e26-a773-61487fa35aff-kube-api-access-xwjjh\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.869010 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.868981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-maas-api-tls\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.869231 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.869214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-metrics-certs\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.869361 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.869347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3e9ab6-e956-4e26-a773-61487fa35aff-openshift-service-ca\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.970739 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.970711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-maas-api-tls\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.970891 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.970765 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-metrics-certs\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.970891 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.970810 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3e9ab6-e956-4e26-a773-61487fa35aff-openshift-service-ca\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.971095 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.970925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjjh\" (UniqueName: \"kubernetes.io/projected/ef3e9ab6-e956-4e26-a773-61487fa35aff-kube-api-access-xwjjh\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.971653 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.971627 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3e9ab6-e956-4e26-a773-61487fa35aff-openshift-service-ca\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.973420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.973396 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-metrics-certs\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.973420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.973407 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-maas-api-tls\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:26.979473 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:26.979445 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjjh\" (UniqueName: \"kubernetes.io/projected/ef3e9ab6-e956-4e26-a773-61487fa35aff-kube-api-access-xwjjh\") pod \"maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:27.067870 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:27.067844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:27.352413 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:27.352373 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj"] Sep 19 12:40:27.352767 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:40:27.352743 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3e9ab6_e956_4e26_a773_61487fa35aff.slice/crio-85308ac9f904459a7355650c5864982bb69cd797b76071bb1e2949786f994bef WatchSource:0}: Error finding container 85308ac9f904459a7355650c5864982bb69cd797b76071bb1e2949786f994bef: Status 404 returned error can't find the container with id 85308ac9f904459a7355650c5864982bb69cd797b76071bb1e2949786f994bef Sep 19 12:40:28.161672 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:28.161639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" event={"ID":"ef3e9ab6-e956-4e26-a773-61487fa35aff","Type":"ContainerStarted","Data":"7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c"} Sep 19 12:40:28.162071 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:28.161875 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:40:28.162071 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:28.161908 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" event={"ID":"ef3e9ab6-e956-4e26-a773-61487fa35aff","Type":"ContainerStarted","Data":"85308ac9f904459a7355650c5864982bb69cd797b76071bb1e2949786f994bef"} Sep 19 12:40:28.181197 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:28.181147 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" podStartSLOduration=1.813606726 podStartE2EDuration="2.181132236s" totalImagesPullingTime="367.52551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:40:28.179272019 +0000 UTC m=+923.786122629" watchObservedRunningTime="2026-09-19 12:40:28.181132236 +0000 UTC m=+923.787982851" Sep 19 12:40:34.174816 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:40:34.174785 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:41:04.860531 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:04.860442 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj"] Sep 19 12:41:04.861148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:04.860873 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" podUID="ef3e9ab6-e956-4e26-a773-61487fa35aff" containerName="maas-api" containerID="cri-o://7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c" gracePeriod=30 Sep 19 12:41:05.110852 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.110784 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:41:05.176525 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.176489 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-metrics-certs\") pod \"ef3e9ab6-e956-4e26-a773-61487fa35aff\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " Sep 19 12:41:05.176787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.176552 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-maas-api-tls\") pod \"ef3e9ab6-e956-4e26-a773-61487fa35aff\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " Sep 19 12:41:05.176787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.176583 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3e9ab6-e956-4e26-a773-61487fa35aff-openshift-service-ca\") pod \"ef3e9ab6-e956-4e26-a773-61487fa35aff\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " Sep 19 12:41:05.176787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.176726 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwjjh\" (UniqueName: \"kubernetes.io/projected/ef3e9ab6-e956-4e26-a773-61487fa35aff-kube-api-access-xwjjh\") pod \"ef3e9ab6-e956-4e26-a773-61487fa35aff\" (UID: \"ef3e9ab6-e956-4e26-a773-61487fa35aff\") " Sep 19 12:41:05.177064 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.177025 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3e9ab6-e956-4e26-a773-61487fa35aff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef3e9ab6-e956-4e26-a773-61487fa35aff" (UID: "ef3e9ab6-e956-4e26-a773-61487fa35aff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:41:05.177213 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.177192 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3e9ab6-e956-4e26-a773-61487fa35aff-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:05.179448 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.179423 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3e9ab6-e956-4e26-a773-61487fa35aff-kube-api-access-xwjjh" (OuterVolumeSpecName: "kube-api-access-xwjjh") pod "ef3e9ab6-e956-4e26-a773-61487fa35aff" (UID: "ef3e9ab6-e956-4e26-a773-61487fa35aff"). InnerVolumeSpecName "kube-api-access-xwjjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:41:05.179727 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.179702 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef3e9ab6-e956-4e26-a773-61487fa35aff" (UID: "ef3e9ab6-e956-4e26-a773-61487fa35aff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:41:05.179830 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.179702 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef3e9ab6-e956-4e26-a773-61487fa35aff" (UID: "ef3e9ab6-e956-4e26-a773-61487fa35aff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:41:05.278310 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.278270 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:05.278310 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.278310 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3e9ab6-e956-4e26-a773-61487fa35aff-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:05.278531 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.278325 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjjh\" (UniqueName: \"kubernetes.io/projected/ef3e9ab6-e956-4e26-a773-61487fa35aff-kube-api-access-xwjjh\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:05.420231 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.420131 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz"] Sep 19 12:41:05.420447 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.420380 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" containerID="cri-o://70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9" gracePeriod=30 Sep 19 12:41:05.424389 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.424358 2585 generic.go:356] "Generic (PLEG): container finished" podID="ef3e9ab6-e956-4e26-a773-61487fa35aff" containerID="7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c" exitCode=0 Sep 19 12:41:05.424571 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.424553 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" event={"ID":"ef3e9ab6-e956-4e26-a773-61487fa35aff","Type":"ContainerDied","Data":"7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c"} Sep 19 12:41:05.424757 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.424602 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" event={"ID":"ef3e9ab6-e956-4e26-a773-61487fa35aff","Type":"ContainerDied","Data":"85308ac9f904459a7355650c5864982bb69cd797b76071bb1e2949786f994bef"} Sep 19 12:41:05.424757 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.424631 2585 scope.go:117] "RemoveContainer" containerID="7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c" Sep 19 12:41:05.424757 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.424652 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj" Sep 19 12:41:05.438248 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.438226 2585 scope.go:117] "RemoveContainer" containerID="7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c" Sep 19 12:41:05.438539 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:41:05.438518 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c\": container with ID starting with 7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c not found: ID does not exist" containerID="7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c" Sep 19 12:41:05.438608 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.438552 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c"} err="failed to get container status \"7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c\": rpc error: code = NotFound desc = could not find container \"7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c\": container with ID starting with 7c28acc3d66afac400f86858431186ae59544e106a54580ac675884d133b605c not found: ID does not exist" Sep 19 12:41:05.460646 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.460619 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj"] Sep 19 12:41:05.466522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:05.466498 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21b2975e-6cc887f689-9mpxj"] Sep 19 12:41:06.582947 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:06.582915 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3e9ab6-e956-4e26-a773-61487fa35aff" path="/var/lib/kubelet/pods/ef3e9ab6-e956-4e26-a773-61487fa35aff/volumes" Sep 19 12:41:10.698316 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.698256 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:41:10.835271 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835238 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7skb\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-kube-api-access-d7skb\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835470 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835296 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-certs\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835470 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835330 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-data\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835470 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835354 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-istio-token\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835470 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835412 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-envoy\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835470 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835436 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-socket\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835527 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14111d82-d34e-4742-9994-df76b8b18400-istio-podinfo\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835553 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14111d82-d34e-4742-9994-df76b8b18400-istiod-ca-cert\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835607 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-credential-socket\") pod \"14111d82-d34e-4742-9994-df76b8b18400\" (UID: \"14111d82-d34e-4742-9994-df76b8b18400\") " Sep 19 12:41:10.835790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835634 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:41:10.835790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835675 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-data" (OuterVolumeSpecName: "istio-data") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:41:10.836106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835787 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:41:10.836106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.835998 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:41:10.836106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.836025 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.836106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.836046 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-data\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.836106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.836060 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-workload-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.836106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.836027 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14111d82-d34e-4742-9994-df76b8b18400-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:41:10.837915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.837893 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:41:10.838054 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.837932 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-istio-token" (OuterVolumeSpecName: "istio-token") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:41:10.838054 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.837999 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14111d82-d34e-4742-9994-df76b8b18400-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 12:41:10.838141 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.838079 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-kube-api-access-d7skb" (OuterVolumeSpecName: "kube-api-access-d7skb") pod "14111d82-d34e-4742-9994-df76b8b18400" (UID: "14111d82-d34e-4742-9994-df76b8b18400"). InnerVolumeSpecName "kube-api-access-d7skb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:41:10.936876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.936842 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-istio-envoy\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.936876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.936878 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14111d82-d34e-4742-9994-df76b8b18400-istio-podinfo\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.937086 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.936893 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14111d82-d34e-4742-9994-df76b8b18400-istiod-ca-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.937086 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.936907 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14111d82-d34e-4742-9994-df76b8b18400-credential-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.937086 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.936922 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7skb\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-kube-api-access-d7skb\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:10.937086 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:10.936935 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14111d82-d34e-4742-9994-df76b8b18400-istio-token\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:41:11.466813 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.466776 2585 generic.go:356] "Generic (PLEG): container finished" podID="14111d82-d34e-4742-9994-df76b8b18400" containerID="70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9" exitCode=0 Sep 19 12:41:11.466990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.466842 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" event={"ID":"14111d82-d34e-4742-9994-df76b8b18400","Type":"ContainerDied","Data":"70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9"} Sep 19 12:41:11.466990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.466847 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" Sep 19 12:41:11.466990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.466912 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz" event={"ID":"14111d82-d34e-4742-9994-df76b8b18400","Type":"ContainerDied","Data":"bd8fc4c2cd67500a116a96ddd9bd65886ac356ee08cbeca7700d07de8f50d03c"} Sep 19 12:41:11.466990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.466933 2585 scope.go:117] "RemoveContainer" containerID="70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9" Sep 19 12:41:11.480449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.480425 2585 scope.go:117] "RemoveContainer" containerID="6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c" Sep 19 12:41:11.489632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.489608 2585 scope.go:117] "RemoveContainer" containerID="70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9" Sep 19 12:41:11.489972 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:41:11.489953 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9\": container with ID starting with 70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9 not found: ID does not exist" containerID="70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9" Sep 19 12:41:11.490049 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.489981 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9"} err="failed to get container status \"70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9\": rpc error: code = NotFound desc = could not find container \"70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9\": container with ID starting with 70be4618614b5aa77433e927eb10d289cc49fdb49cdedb871e20d832a94ae2c9 not found: ID does not exist" Sep 19 12:41:11.490049 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.489999 2585 scope.go:117] "RemoveContainer" containerID="6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c" Sep 19 12:41:11.490273 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:41:11.490253 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c\": container with ID starting with 6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c not found: ID does not exist" containerID="6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c" Sep 19 12:41:11.490338 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.490278 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c"} err="failed to get container status \"6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c\": rpc error: code = NotFound desc = could not find container \"6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c\": container with ID starting with 6f3109aa01fc9ee4ffbc34d525edd8d3c708b54f40df2e0112df3e025c2a206c not found: ID does not exist" Sep 19 12:41:11.492407 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.492384 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz"] Sep 19 12:41:11.498215 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:11.498191 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-38aa0a-openshift-default-656d955f98-m45jz"] Sep 19 12:41:12.581432 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:41:12.581389 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14111d82-d34e-4742-9994-df76b8b18400" path="/var/lib/kubelet/pods/14111d82-d34e-4742-9994-df76b8b18400/volumes" Sep 19 12:42:04.332540 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.332506 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx"] Sep 19 12:42:04.333062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333048 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" Sep 19 12:42:04.333107 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333064 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" Sep 19 12:42:04.333107 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333082 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3e9ab6-e956-4e26-a773-61487fa35aff" containerName="maas-api" Sep 19 12:42:04.333107 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333087 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3e9ab6-e956-4e26-a773-61487fa35aff" containerName="maas-api" Sep 19 12:42:04.333208 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333173 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" Sep 19 12:42:04.333208 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333183 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" Sep 19 12:42:04.333208 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.333190 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3e9ab6-e956-4e26-a773-61487fa35aff" containerName="maas-api" Sep 19 12:42:04.336515 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.336499 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.338879 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.338859 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-04a668\"/\"default-dockercfg-lnbhr\"" Sep 19 12:42:04.339109 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.339092 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-04a668\"/\"openshift-service-ca.crt\"" Sep 19 12:42:04.339932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.339910 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-04a668\"/\"kube-root-ca.crt\"" Sep 19 12:42:04.340067 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.339991 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-04a668\"/\"premium-w4-04a668-kserve-self-signed-certs\"" Sep 19 12:42:04.343862 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.343839 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx"] Sep 19 12:42:04.419565 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.419536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a153fee6-a54e-4158-8a19-c981f0d2e54d-tls-certs\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.420151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.419625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsnm\" (UniqueName: \"kubernetes.io/projected/a153fee6-a54e-4158-8a19-c981f0d2e54d-kube-api-access-5gsnm\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.420281 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.420216 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-dshm\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.420345 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.420284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-tmp-dir\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.420435 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.420409 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-model-cache\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.420666 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.420636 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-home\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.521600 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.521567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsnm\" (UniqueName: \"kubernetes.io/projected/a153fee6-a54e-4158-8a19-c981f0d2e54d-kube-api-access-5gsnm\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.521819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.521610 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-dshm\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.521819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.521633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-tmp-dir\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.521819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.521656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-model-cache\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.522003 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.521829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-home\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.522147 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.522122 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-tmp-dir\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.522241 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.522198 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-model-cache\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.522241 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.522212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a153fee6-a54e-4158-8a19-c981f0d2e54d-tls-certs\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.522532 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.522509 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-home\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.524314 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.524285 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-dshm\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.525238 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.525217 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a153fee6-a54e-4158-8a19-c981f0d2e54d-tls-certs\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.530544 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.530513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsnm\" (UniqueName: \"kubernetes.io/projected/a153fee6-a54e-4158-8a19-c981f0d2e54d-kube-api-access-5gsnm\") pod \"premium-w4-04a668-kserve-7565dbb78c-bvsrx\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.647772 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.647710 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:04.923972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:04.923943 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx"] Sep 19 12:42:04.925303 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:42:04.925278 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda153fee6_a54e_4158_8a19_c981f0d2e54d.slice/crio-8bdc802ae2cc05534c76d95f54d750a22b646a1b140d737685128efdcff376c3 WatchSource:0}: Error finding container 8bdc802ae2cc05534c76d95f54d750a22b646a1b140d737685128efdcff376c3: Status 404 returned error can't find the container with id 8bdc802ae2cc05534c76d95f54d750a22b646a1b140d737685128efdcff376c3 Sep 19 12:42:05.837090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:05.837053 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" event={"ID":"a153fee6-a54e-4158-8a19-c981f0d2e54d","Type":"ContainerStarted","Data":"cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c"} Sep 19 12:42:05.837528 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:05.837099 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" event={"ID":"a153fee6-a54e-4158-8a19-c981f0d2e54d","Type":"ContainerStarted","Data":"8bdc802ae2cc05534c76d95f54d750a22b646a1b140d737685128efdcff376c3"} Sep 19 12:42:05.856246 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:05.856193 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" podStartSLOduration=1.856179142 podStartE2EDuration="1.856179142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:42:05.854477674 +0000 UTC m=+1021.461328280" watchObservedRunningTime="2026-09-19 12:42:05.856179142 +0000 UTC m=+1021.463029722" Sep 19 12:42:14.648005 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:14.647972 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:14.648005 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:14.648012 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:14.660735 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:14.660680 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:14.911179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:14.911103 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:42:49.430357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.430261 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm"] Sep 19 12:42:49.431070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.431048 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" Sep 19 12:42:49.431070 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.431071 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="14111d82-d34e-4742-9994-df76b8b18400" containerName="istio-proxy" Sep 19 12:42:49.434815 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.434794 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.438201 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.438072 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f9a7df96-serving-cert\"" Sep 19 12:42:49.446294 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.446269 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm"] Sep 19 12:42:49.549744 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.549675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6218f-2038-4fd0-919b-1267c54a4c89-openshift-service-ca\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.549915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.549849 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-metrics-certs\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.549915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.549899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqsm\" (UniqueName: \"kubernetes.io/projected/14b6218f-2038-4fd0-919b-1267c54a4c89-kube-api-access-xqqsm\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.550020 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.549978 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-maas-api-tls\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.651323 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.651290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6218f-2038-4fd0-919b-1267c54a4c89-openshift-service-ca\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.651485 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.651390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-metrics-certs\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.651485 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.651437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqsm\" (UniqueName: \"kubernetes.io/projected/14b6218f-2038-4fd0-919b-1267c54a4c89-kube-api-access-xqqsm\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.651485 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.651472 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-maas-api-tls\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.652077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.652056 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6218f-2038-4fd0-919b-1267c54a4c89-openshift-service-ca\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.654164 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.654135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-metrics-certs\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.654282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.654143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-maas-api-tls\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.660060 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.660041 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqsm\" (UniqueName: \"kubernetes.io/projected/14b6218f-2038-4fd0-919b-1267c54a4c89-kube-api-access-xqqsm\") pod \"maas-api-e2e-mt-f9a7df96-76c9597758-slxmm\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:49.749990 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:49.749906 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:50.037073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:50.037043 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm"] Sep 19 12:42:50.038748 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:42:50.038716 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b6218f_2038_4fd0_919b_1267c54a4c89.slice/crio-8aeff577e8678b44fec83c1e5fa9ce32c77fae3c8d14d3f2e74e3212a78f9c32 WatchSource:0}: Error finding container 8aeff577e8678b44fec83c1e5fa9ce32c77fae3c8d14d3f2e74e3212a78f9c32: Status 404 returned error can't find the container with id 8aeff577e8678b44fec83c1e5fa9ce32c77fae3c8d14d3f2e74e3212a78f9c32 Sep 19 12:42:50.040365 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:50.040344 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 12:42:50.136406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:50.136366 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" event={"ID":"14b6218f-2038-4fd0-919b-1267c54a4c89","Type":"ContainerStarted","Data":"8aeff577e8678b44fec83c1e5fa9ce32c77fae3c8d14d3f2e74e3212a78f9c32"} Sep 19 12:42:51.144321 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:51.144286 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" event={"ID":"14b6218f-2038-4fd0-919b-1267c54a4c89","Type":"ContainerStarted","Data":"5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d"} Sep 19 12:42:51.144733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:51.144343 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:42:51.163817 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:51.163770 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" podStartSLOduration=1.5254273839999999 podStartE2EDuration="2.163755704s" totalImagesPullingTime="638.32832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:42:51.160272578 +0000 UTC m=+1066.767123182" watchObservedRunningTime="2026-09-19 12:42:51.163755704 +0000 UTC m=+1066.770606282" Sep 19 12:42:57.155593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:42:57.155559 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:43:10.121626 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.121591 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl"] Sep 19 12:43:10.125635 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.125613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.129484 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.129463 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b4e91c\"/\"openshift-service-ca.crt\"" Sep 19 12:43:10.129627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.129480 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b4e91c\"/\"explicit-model-b4e91c-kserve-self-signed-certs\"" Sep 19 12:43:10.129627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.129506 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b4e91c\"/\"kube-root-ca.crt\"" Sep 19 12:43:10.129627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.129517 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b4e91c\"/\"default-dockercfg-75xc6\"" Sep 19 12:43:10.135632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.135608 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl"] Sep 19 12:43:10.242825 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.242790 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-model-cache\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.242993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.242870 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-home\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.242993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.242893 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdv2s\" (UniqueName: \"kubernetes.io/projected/78b666f9-5206-4d80-92b3-88ec09ec21ed-kube-api-access-bdv2s\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.242993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.242957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-dshm\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.242993 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.242983 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-tmp-dir\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.243157 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.243005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b666f9-5206-4d80-92b3-88ec09ec21ed-tls-certs\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.343670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.343635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-home\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.343670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.343671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdv2s\" (UniqueName: \"kubernetes.io/projected/78b666f9-5206-4d80-92b3-88ec09ec21ed-kube-api-access-bdv2s\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.343942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.343756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-dshm\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.343942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.343795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-tmp-dir\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.343942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.343830 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b666f9-5206-4d80-92b3-88ec09ec21ed-tls-certs\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.343942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.343885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-model-cache\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.344196 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.344097 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-home\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.344237 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.344193 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-tmp-dir\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.344275 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.344245 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-model-cache\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.346284 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.346261 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-dshm\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.346449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.346433 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b666f9-5206-4d80-92b3-88ec09ec21ed-tls-certs\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.351514 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.351491 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdv2s\" (UniqueName: \"kubernetes.io/projected/78b666f9-5206-4d80-92b3-88ec09ec21ed-kube-api-access-bdv2s\") pod \"explicit-model-b4e91c-kserve-65446d7bfc-95rkl\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.436249 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.436160 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:10.716964 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:10.716924 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl"] Sep 19 12:43:10.717650 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:43:10.717626 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78b666f9_5206_4d80_92b3_88ec09ec21ed.slice/crio-a0b2faf494369b2586a70c52ac751a773b8904e3191ed2a290fc71c1f1a6786e WatchSource:0}: Error finding container a0b2faf494369b2586a70c52ac751a773b8904e3191ed2a290fc71c1f1a6786e: Status 404 returned error can't find the container with id a0b2faf494369b2586a70c52ac751a773b8904e3191ed2a290fc71c1f1a6786e Sep 19 12:43:11.279867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:11.279793 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" event={"ID":"78b666f9-5206-4d80-92b3-88ec09ec21ed","Type":"ContainerStarted","Data":"7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3"} Sep 19 12:43:11.280303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:11.279883 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" event={"ID":"78b666f9-5206-4d80-92b3-88ec09ec21ed","Type":"ContainerStarted","Data":"a0b2faf494369b2586a70c52ac751a773b8904e3191ed2a290fc71c1f1a6786e"} Sep 19 12:43:11.296301 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:11.296251 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" podStartSLOduration=1.296232263 podStartE2EDuration="1.296232263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:43:11.295029346 +0000 UTC m=+1086.901879943" watchObservedRunningTime="2026-09-19 12:43:11.296232263 +0000 UTC m=+1086.903082843" Sep 19 12:43:20.436708 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:20.436646 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:20.437201 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:20.436734 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:20.449629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:20.449604 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:21.363541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:21.363504 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:43:33.982869 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:33.982607 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm"] Sep 19 12:43:33.983521 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:33.982941 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" podUID="14b6218f-2038-4fd0-919b-1267c54a4c89" containerName="maas-api" containerID="cri-o://5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d" gracePeriod=30 Sep 19 12:43:34.240457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.240402 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:43:34.292745 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.292717 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqqsm\" (UniqueName: \"kubernetes.io/projected/14b6218f-2038-4fd0-919b-1267c54a4c89-kube-api-access-xqqsm\") pod \"14b6218f-2038-4fd0-919b-1267c54a4c89\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " Sep 19 12:43:34.292931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.292778 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6218f-2038-4fd0-919b-1267c54a4c89-openshift-service-ca\") pod \"14b6218f-2038-4fd0-919b-1267c54a4c89\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " Sep 19 12:43:34.292931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.292833 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-maas-api-tls\") pod \"14b6218f-2038-4fd0-919b-1267c54a4c89\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " Sep 19 12:43:34.293051 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.292948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-metrics-certs\") pod \"14b6218f-2038-4fd0-919b-1267c54a4c89\" (UID: \"14b6218f-2038-4fd0-919b-1267c54a4c89\") " Sep 19 12:43:34.293179 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.293155 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b6218f-2038-4fd0-919b-1267c54a4c89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14b6218f-2038-4fd0-919b-1267c54a4c89" (UID: "14b6218f-2038-4fd0-919b-1267c54a4c89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:43:34.293372 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.293351 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6218f-2038-4fd0-919b-1267c54a4c89-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:43:34.295393 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.295368 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b6218f-2038-4fd0-919b-1267c54a4c89-kube-api-access-xqqsm" (OuterVolumeSpecName: "kube-api-access-xqqsm") pod "14b6218f-2038-4fd0-919b-1267c54a4c89" (UID: "14b6218f-2038-4fd0-919b-1267c54a4c89"). InnerVolumeSpecName "kube-api-access-xqqsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:43:34.295518 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.295423 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14b6218f-2038-4fd0-919b-1267c54a4c89" (UID: "14b6218f-2038-4fd0-919b-1267c54a4c89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:43:34.295642 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.295622 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14b6218f-2038-4fd0-919b-1267c54a4c89" (UID: "14b6218f-2038-4fd0-919b-1267c54a4c89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:43:34.394361 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.394316 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:43:34.394361 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.394353 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6218f-2038-4fd0-919b-1267c54a4c89-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:43:34.394561 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.394367 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqqsm\" (UniqueName: \"kubernetes.io/projected/14b6218f-2038-4fd0-919b-1267c54a4c89-kube-api-access-xqqsm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:43:34.446408 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.446375 2585 generic.go:356] "Generic (PLEG): container finished" podID="14b6218f-2038-4fd0-919b-1267c54a4c89" containerID="5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d" exitCode=0 Sep 19 12:43:34.446408 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.446421 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" event={"ID":"14b6218f-2038-4fd0-919b-1267c54a4c89","Type":"ContainerDied","Data":"5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d"} Sep 19 12:43:34.446631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.446465 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" event={"ID":"14b6218f-2038-4fd0-919b-1267c54a4c89","Type":"ContainerDied","Data":"8aeff577e8678b44fec83c1e5fa9ce32c77fae3c8d14d3f2e74e3212a78f9c32"} Sep 19 12:43:34.446631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.446485 2585 scope.go:117] "RemoveContainer" containerID="5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d" Sep 19 12:43:34.446631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.446420 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm" Sep 19 12:43:34.457320 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.457305 2585 scope.go:117] "RemoveContainer" containerID="5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d" Sep 19 12:43:34.457627 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:43:34.457601 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d\": container with ID starting with 5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d not found: ID does not exist" containerID="5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d" Sep 19 12:43:34.457696 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.457658 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d"} err="failed to get container status \"5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d\": rpc error: code = NotFound desc = could not find container \"5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d\": container with ID starting with 5b741dbbeeac4a0bad2b5eec41a42d0e756622b0edee414cb3e2631cb555aa6d not found: ID does not exist" Sep 19 12:43:34.468108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.468079 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm"] Sep 19 12:43:34.472448 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.472425 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f9a7df96-76c9597758-slxmm"] Sep 19 12:43:34.581216 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:34.581177 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b6218f-2038-4fd0-919b-1267c54a4c89" path="/var/lib/kubelet/pods/14b6218f-2038-4fd0-919b-1267c54a4c89/volumes" Sep 19 12:43:58.092962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:43:58.092929 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl"] Sep 19 12:44:01.333890 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.333851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb"] Sep 19 12:44:01.334387 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.334374 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b6218f-2038-4fd0-919b-1267c54a4c89" containerName="maas-api" Sep 19 12:44:01.334387 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.334388 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b6218f-2038-4fd0-919b-1267c54a4c89" containerName="maas-api" Sep 19 12:44:01.334492 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.334483 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b6218f-2038-4fd0-919b-1267c54a4c89" containerName="maas-api" Sep 19 12:44:01.338027 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.338004 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.340489 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.340462 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2a487c-serving-cert\"" Sep 19 12:44:01.348350 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.347671 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb"] Sep 19 12:44:01.484644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.484604 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67a11cc-ba15-45d8-8fcb-740a2b382608-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.484895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.484672 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.484972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.484886 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkz2c\" (UniqueName: \"kubernetes.io/projected/f67a11cc-ba15-45d8-8fcb-740a2b382608-kube-api-access-wkz2c\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.484972 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.484937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.586541 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.586450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.586740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.586540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkz2c\" (UniqueName: \"kubernetes.io/projected/f67a11cc-ba15-45d8-8fcb-740a2b382608-kube-api-access-wkz2c\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.586740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.586568 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.586740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.586590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67a11cc-ba15-45d8-8fcb-740a2b382608-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.586740 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:44:01.586721 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-serving-cert: secret "maas-api-e2e-shared-a-w2-2a487c-serving-cert" not found Sep 19 12:44:01.586911 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:44:01.586789 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls podName:f67a11cc-ba15-45d8-8fcb-740a2b382608 nodeName:}" failed. No retries permitted until 2026-09-19 12:44:02.086767117 +0000 UTC m=+1137.693617676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls") pod "maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" (UID: "f67a11cc-ba15-45d8-8fcb-740a2b382608") : secret "maas-api-e2e-shared-a-w2-2a487c-serving-cert" not found Sep 19 12:44:01.587285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.587266 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67a11cc-ba15-45d8-8fcb-740a2b382608-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.589218 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.589196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:01.594921 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:01.594898 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkz2c\" (UniqueName: \"kubernetes.io/projected/f67a11cc-ba15-45d8-8fcb-740a2b382608-kube-api-access-wkz2c\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:02.092243 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:02.092196 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:02.095474 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:02.095438 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:02.252909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:02.252865 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:02.534181 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:02.534153 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb"] Sep 19 12:44:02.536133 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:44:02.536102 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67a11cc_ba15_45d8_8fcb_740a2b382608.slice/crio-45ef8414a636d218d76186bf9eae9961493bec33ed9d4d658315c1f6e999f6a1 WatchSource:0}: Error finding container 45ef8414a636d218d76186bf9eae9961493bec33ed9d4d658315c1f6e999f6a1: Status 404 returned error can't find the container with id 45ef8414a636d218d76186bf9eae9961493bec33ed9d4d658315c1f6e999f6a1 Sep 19 12:44:02.652886 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:02.652851 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" event={"ID":"f67a11cc-ba15-45d8-8fcb-740a2b382608","Type":"ContainerStarted","Data":"45ef8414a636d218d76186bf9eae9961493bec33ed9d4d658315c1f6e999f6a1"} Sep 19 12:44:03.661169 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:03.661126 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" event={"ID":"f67a11cc-ba15-45d8-8fcb-740a2b382608","Type":"ContainerStarted","Data":"49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890"} Sep 19 12:44:03.661611 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:03.661210 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:03.680004 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:03.679943 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" podStartSLOduration=2.101454963 podStartE2EDuration="2.67992573s" totalImagesPullingTime="578.470767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:44:03.677286957 +0000 UTC m=+1139.284137583" watchObservedRunningTime="2026-09-19 12:44:03.67992573 +0000 UTC m=+1139.286776309" Sep 19 12:44:09.676904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:09.676871 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:44:13.118129 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.118085 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" podUID="78b666f9-5206-4d80-92b3-88ec09ec21ed" containerName="main" containerID="cri-o://7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3" gracePeriod=45 Sep 19 12:44:13.387419 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.387354 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:44:13.413641 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.413604 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-home\") pod \"78b666f9-5206-4d80-92b3-88ec09ec21ed\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " Sep 19 12:44:13.413857 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.413673 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b666f9-5206-4d80-92b3-88ec09ec21ed-tls-certs\") pod \"78b666f9-5206-4d80-92b3-88ec09ec21ed\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " Sep 19 12:44:13.413857 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.413724 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-tmp-dir\") pod \"78b666f9-5206-4d80-92b3-88ec09ec21ed\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " Sep 19 12:44:13.413857 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.413759 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdv2s\" (UniqueName: \"kubernetes.io/projected/78b666f9-5206-4d80-92b3-88ec09ec21ed-kube-api-access-bdv2s\") pod \"78b666f9-5206-4d80-92b3-88ec09ec21ed\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " Sep 19 12:44:13.413857 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.413834 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-model-cache\") pod \"78b666f9-5206-4d80-92b3-88ec09ec21ed\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " Sep 19 12:44:13.414094 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.413898 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-dshm\") pod \"78b666f9-5206-4d80-92b3-88ec09ec21ed\" (UID: \"78b666f9-5206-4d80-92b3-88ec09ec21ed\") " Sep 19 12:44:13.414631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.414604 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-home" (OuterVolumeSpecName: "home") pod "78b666f9-5206-4d80-92b3-88ec09ec21ed" (UID: "78b666f9-5206-4d80-92b3-88ec09ec21ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:44:13.414835 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.414810 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78b666f9-5206-4d80-92b3-88ec09ec21ed" (UID: "78b666f9-5206-4d80-92b3-88ec09ec21ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:44:13.414932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.414736 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "78b666f9-5206-4d80-92b3-88ec09ec21ed" (UID: "78b666f9-5206-4d80-92b3-88ec09ec21ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:44:13.418996 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.418953 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b666f9-5206-4d80-92b3-88ec09ec21ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78b666f9-5206-4d80-92b3-88ec09ec21ed" (UID: "78b666f9-5206-4d80-92b3-88ec09ec21ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:44:13.419263 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.419232 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b666f9-5206-4d80-92b3-88ec09ec21ed-kube-api-access-bdv2s" (OuterVolumeSpecName: "kube-api-access-bdv2s") pod "78b666f9-5206-4d80-92b3-88ec09ec21ed" (UID: "78b666f9-5206-4d80-92b3-88ec09ec21ed"). InnerVolumeSpecName "kube-api-access-bdv2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:44:13.421379 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.421349 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-dshm" (OuterVolumeSpecName: "dshm") pod "78b666f9-5206-4d80-92b3-88ec09ec21ed" (UID: "78b666f9-5206-4d80-92b3-88ec09ec21ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:44:13.515514 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.515478 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-home\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:44:13.515514 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.515511 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b666f9-5206-4d80-92b3-88ec09ec21ed-tls-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:44:13.515751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.515524 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-tmp-dir\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:44:13.515751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.515536 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdv2s\" (UniqueName: \"kubernetes.io/projected/78b666f9-5206-4d80-92b3-88ec09ec21ed-kube-api-access-bdv2s\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:44:13.515751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.515545 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-model-cache\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:44:13.515751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.515553 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b666f9-5206-4d80-92b3-88ec09ec21ed-dshm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:44:13.737740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.737631 2585 generic.go:356] "Generic (PLEG): container finished" podID="78b666f9-5206-4d80-92b3-88ec09ec21ed" containerID="7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3" exitCode=0 Sep 19 12:44:13.737740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.737719 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" event={"ID":"78b666f9-5206-4d80-92b3-88ec09ec21ed","Type":"ContainerDied","Data":"7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3"} Sep 19 12:44:13.737942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.737767 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" event={"ID":"78b666f9-5206-4d80-92b3-88ec09ec21ed","Type":"ContainerDied","Data":"a0b2faf494369b2586a70c52ac751a773b8904e3191ed2a290fc71c1f1a6786e"} Sep 19 12:44:13.737942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.737783 2585 scope.go:117] "RemoveContainer" containerID="7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3" Sep 19 12:44:13.737942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.737727 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl" Sep 19 12:44:13.748339 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.748319 2585 scope.go:117] "RemoveContainer" containerID="7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3" Sep 19 12:44:13.748662 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:44:13.748642 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3\": container with ID starting with 7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3 not found: ID does not exist" containerID="7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3" Sep 19 12:44:13.748751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.748672 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3"} err="failed to get container status \"7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3\": rpc error: code = NotFound desc = could not find container \"7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3\": container with ID starting with 7d42e3e317f84c495c9b308a0381044aaab2ccbc3e63fb6d289d11b99fd2e9b3 not found: ID does not exist" Sep 19 12:44:13.761588 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.761559 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl"] Sep 19 12:44:13.771154 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:13.771060 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b4e91c/explicit-model-b4e91c-kserve-65446d7bfc-95rkl"] Sep 19 12:44:14.581826 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:14.581797 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b666f9-5206-4d80-92b3-88ec09ec21ed" path="/var/lib/kubelet/pods/78b666f9-5206-4d80-92b3-88ec09ec21ed/volumes" Sep 19 12:44:35.115240 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.115198 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx"] Sep 19 12:44:35.115788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.115773 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b666f9-5206-4d80-92b3-88ec09ec21ed" containerName="main" Sep 19 12:44:35.115788 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.115790 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b666f9-5206-4d80-92b3-88ec09ec21ed" containerName="main" Sep 19 12:44:35.115934 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.115899 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b666f9-5206-4d80-92b3-88ec09ec21ed" containerName="main" Sep 19 12:44:35.118939 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.118920 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.121418 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.121394 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-04a668\"/\"e2e-conflict-a-3ed226d2-kserve-self-signed-certs\"" Sep 19 12:44:35.129457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.129434 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx"] Sep 19 12:44:35.171761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.171416 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-dshm\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.171761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.171512 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kg9\" (UniqueName: \"kubernetes.io/projected/582de267-73ae-458f-8d27-34667310ffd8-kube-api-access-f9kg9\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.171761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.171578 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-model-cache\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.171761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.171606 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-home\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.171761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.171671 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-tmp-dir\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.171761 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.171710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582de267-73ae-458f-8d27-34667310ffd8-tls-certs\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272440 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-model-cache\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-home\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272526 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-tmp-dir\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582de267-73ae-458f-8d27-34667310ffd8-tls-certs\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272661 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272613 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-dshm\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272740 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kg9\" (UniqueName: \"kubernetes.io/projected/582de267-73ae-458f-8d27-34667310ffd8-kube-api-access-f9kg9\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.272895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.272880 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-model-cache\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.273155 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.273131 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-tmp-dir\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.274081 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.274053 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-home\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.276636 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.276606 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582de267-73ae-458f-8d27-34667310ffd8-tls-certs\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.276858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.276838 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-dshm\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.282324 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.282299 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kg9\" (UniqueName: \"kubernetes.io/projected/582de267-73ae-458f-8d27-34667310ffd8-kube-api-access-f9kg9\") pod \"e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.429391 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.429300 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:35.736550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.736520 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx"] Sep 19 12:44:35.738536 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:44:35.738505 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod582de267_73ae_458f_8d27_34667310ffd8.slice/crio-e0d776c6c8278814eb1c604626524d8e0731a6d6cc324ab7d39e26f75134c381 WatchSource:0}: Error finding container e0d776c6c8278814eb1c604626524d8e0731a6d6cc324ab7d39e26f75134c381: Status 404 returned error can't find the container with id e0d776c6c8278814eb1c604626524d8e0731a6d6cc324ab7d39e26f75134c381 Sep 19 12:44:35.893241 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.893203 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" event={"ID":"582de267-73ae-458f-8d27-34667310ffd8","Type":"ContainerStarted","Data":"61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963"} Sep 19 12:44:35.893241 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.893252 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" event={"ID":"582de267-73ae-458f-8d27-34667310ffd8","Type":"ContainerStarted","Data":"e0d776c6c8278814eb1c604626524d8e0731a6d6cc324ab7d39e26f75134c381"} Sep 19 12:44:35.917331 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:35.917274 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" podStartSLOduration=0.917257299 podStartE2EDuration="917.257299ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:44:35.916115739 +0000 UTC m=+1171.522966341" watchObservedRunningTime="2026-09-19 12:44:35.917257299 +0000 UTC m=+1171.524107877" Sep 19 12:44:36.256657 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.256622 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56"] Sep 19 12:44:36.260745 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.260725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.263951 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.263924 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5068d9-serving-cert\"" Sep 19 12:44:36.275934 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.274145 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56"] Sep 19 12:44:36.282428 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.282384 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70686a87-7638-4a87-bb58-fbfdac6a6b6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.282621 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.282505 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.282621 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.282549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.282803 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.282660 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnkh\" (UniqueName: \"kubernetes.io/projected/70686a87-7638-4a87-bb58-fbfdac6a6b6a-kube-api-access-ngnkh\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.383336 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.383301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70686a87-7638-4a87-bb58-fbfdac6a6b6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.383522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.383421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.383522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.383468 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.383633 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.383518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnkh\" (UniqueName: \"kubernetes.io/projected/70686a87-7638-4a87-bb58-fbfdac6a6b6a-kube-api-access-ngnkh\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.383633 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:44:36.383571 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-serving-cert: secret "maas-api-e2e-shared-b-w2-5068d9-serving-cert" not found Sep 19 12:44:36.383769 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:44:36.383640 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls podName:70686a87-7638-4a87-bb58-fbfdac6a6b6a nodeName:}" failed. No retries permitted until 2026-09-19 12:44:36.88362173 +0000 UTC m=+1172.490472300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls") pod "maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" (UID: "70686a87-7638-4a87-bb58-fbfdac6a6b6a") : secret "maas-api-e2e-shared-b-w2-5068d9-serving-cert" not found Sep 19 12:44:36.384140 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.384119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70686a87-7638-4a87-bb58-fbfdac6a6b6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.386266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.386236 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.392520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.392496 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnkh\" (UniqueName: \"kubernetes.io/projected/70686a87-7638-4a87-bb58-fbfdac6a6b6a-kube-api-access-ngnkh\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.888506 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.888474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:36.892067 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:36.892042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:37.181152 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:37.181052 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:37.478008 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:37.477970 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56"] Sep 19 12:44:37.480541 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:44:37.480500 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70686a87_7638_4a87_bb58_fbfdac6a6b6a.slice/crio-bdb83ee34b77561a3687822d5f07619888aafde40baa41d946290ad9a81947cf WatchSource:0}: Error finding container bdb83ee34b77561a3687822d5f07619888aafde40baa41d946290ad9a81947cf: Status 404 returned error can't find the container with id bdb83ee34b77561a3687822d5f07619888aafde40baa41d946290ad9a81947cf Sep 19 12:44:37.909220 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:37.909175 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" event={"ID":"70686a87-7638-4a87-bb58-fbfdac6a6b6a","Type":"ContainerStarted","Data":"d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d"} Sep 19 12:44:37.909413 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:37.909248 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:37.909413 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:37.909273 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" event={"ID":"70686a87-7638-4a87-bb58-fbfdac6a6b6a","Type":"ContainerStarted","Data":"bdb83ee34b77561a3687822d5f07619888aafde40baa41d946290ad9a81947cf"} Sep 19 12:44:37.927905 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:37.927842 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" podStartSLOduration=1.63449396 podStartE2EDuration="1.927822611s" totalImagesPullingTime="293.328651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:44:37.926500562 +0000 UTC m=+1173.533351162" watchObservedRunningTime="2026-09-19 12:44:37.927822611 +0000 UTC m=+1173.534673190" Sep 19 12:44:43.922488 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:43.922449 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:44:45.429546 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:45.429507 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:45.430031 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:45.429574 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:45.442047 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:45.442020 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:44:45.977746 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:44:45.977714 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:45:02.792547 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:02.792508 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-b7796d"] Sep 19 12:45:02.796742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:02.796720 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:02.799305 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:02.799278 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pjdxd\"" Sep 19 12:45:02.801975 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:02.801949 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-b7796d"] Sep 19 12:45:02.983368 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:02.983317 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvz76\" (UniqueName: \"kubernetes.io/projected/0b8cf978-733c-42b4-b25a-21b0d9b99299-kube-api-access-xvz76\") pod \"test-curl-7564-b7796d\" (UID: \"0b8cf978-733c-42b4-b25a-21b0d9b99299\") " pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:03.084710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:03.084598 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvz76\" (UniqueName: \"kubernetes.io/projected/0b8cf978-733c-42b4-b25a-21b0d9b99299-kube-api-access-xvz76\") pod \"test-curl-7564-b7796d\" (UID: \"0b8cf978-733c-42b4-b25a-21b0d9b99299\") " pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:03.093067 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:03.093028 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvz76\" (UniqueName: \"kubernetes.io/projected/0b8cf978-733c-42b4-b25a-21b0d9b99299-kube-api-access-xvz76\") pod \"test-curl-7564-b7796d\" (UID: \"0b8cf978-733c-42b4-b25a-21b0d9b99299\") " pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:03.136108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:03.136061 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:03.415229 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:03.415197 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-b7796d"] Sep 19 12:45:03.416510 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:45:03.416483 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8cf978_733c_42b4_b25a_21b0d9b99299.slice/crio-ebcfa2d759db4033eba678121bc41c38393e6aafee1866f9cab0e33d97dd7a31 WatchSource:0}: Error finding container ebcfa2d759db4033eba678121bc41c38393e6aafee1866f9cab0e33d97dd7a31: Status 404 returned error can't find the container with id ebcfa2d759db4033eba678121bc41c38393e6aafee1866f9cab0e33d97dd7a31 Sep 19 12:45:04.092248 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.092215 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-b7796d" event={"ID":"0b8cf978-733c-42b4-b25a-21b0d9b99299","Type":"ContainerStarted","Data":"adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e"} Sep 19 12:45:04.092737 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.092266 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-b7796d" event={"ID":"0b8cf978-733c-42b4-b25a-21b0d9b99299","Type":"ContainerStarted","Data":"ebcfa2d759db4033eba678121bc41c38393e6aafee1866f9cab0e33d97dd7a31"} Sep 19 12:45:04.107148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.107090 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-b7796d" podStartSLOduration=2.107073575 podStartE2EDuration="2.107073575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:45:04.106003066 +0000 UTC m=+1199.712853644" watchObservedRunningTime="2026-09-19 12:45:04.107073575 +0000 UTC m=+1199.713924153" Sep 19 12:45:04.708573 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.708533 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-b7796d"] Sep 19 12:45:04.716625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.716589 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-b7796d"] Sep 19 12:45:04.852257 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.852215 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-c6c5d7"] Sep 19 12:45:04.852839 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.852821 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" containerName="test-curl-7564-b7796d" Sep 19 12:45:04.852892 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.852841 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" containerName="test-curl-7564-b7796d" Sep 19 12:45:04.852946 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.852936 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" containerName="test-curl-7564-b7796d" Sep 19 12:45:04.856398 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.856380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:04.874939 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.874906 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-c6c5d7"] Sep 19 12:45:04.903742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:04.903707 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m89l\" (UniqueName: \"kubernetes.io/projected/478c59e6-fd7a-4730-8161-86a770f3f6b1-kube-api-access-5m89l\") pod \"test-curl-7564-c6c5d7\" (UID: \"478c59e6-fd7a-4730-8161-86a770f3f6b1\") " pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:05.004956 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:05.004918 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m89l\" (UniqueName: \"kubernetes.io/projected/478c59e6-fd7a-4730-8161-86a770f3f6b1-kube-api-access-5m89l\") pod \"test-curl-7564-c6c5d7\" (UID: \"478c59e6-fd7a-4730-8161-86a770f3f6b1\") " pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:05.015853 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:05.015821 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m89l\" (UniqueName: \"kubernetes.io/projected/478c59e6-fd7a-4730-8161-86a770f3f6b1-kube-api-access-5m89l\") pod \"test-curl-7564-c6c5d7\" (UID: \"478c59e6-fd7a-4730-8161-86a770f3f6b1\") " pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:05.099028 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:05.098993 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-b7796d" container="test-curl-7564-b7796d" Sep 19 12:45:05.166566 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:05.166463 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:05.449185 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:05.449155 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-c6c5d7"] Sep 19 12:45:05.450515 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:45:05.450487 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478c59e6_fd7a_4730_8161_86a770f3f6b1.slice/crio-d02661e122e042c75391e504e2dff6ae04bd50e9454b919011ff15d54006c6ed WatchSource:0}: Error finding container d02661e122e042c75391e504e2dff6ae04bd50e9454b919011ff15d54006c6ed: Status 404 returned error can't find the container with id d02661e122e042c75391e504e2dff6ae04bd50e9454b919011ff15d54006c6ed Sep 19 12:45:06.107417 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.107373 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-c6c5d7" event={"ID":"478c59e6-fd7a-4730-8161-86a770f3f6b1","Type":"ContainerStarted","Data":"c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029"} Sep 19 12:45:06.107417 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.107425 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-c6c5d7" event={"ID":"478c59e6-fd7a-4730-8161-86a770f3f6b1","Type":"ContainerStarted","Data":"d02661e122e042c75391e504e2dff6ae04bd50e9454b919011ff15d54006c6ed"} Sep 19 12:45:06.108040 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.107517 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-b7796d" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" containerName="test-curl-7564-b7796d" containerID="cri-o://adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e" gracePeriod=2 Sep 19 12:45:06.126460 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:06.126425 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-b7796d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:06.127178 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.127137 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-c6c5d7" podStartSLOduration=2.127123598 podStartE2EDuration="2.127123598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:45:06.124363795 +0000 UTC m=+1201.731214373" watchObservedRunningTime="2026-09-19 12:45:06.127123598 +0000 UTC m=+1201.733974176" Sep 19 12:45:06.128204 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:06.128184 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-b7796d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:06.578754 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:06.578703 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-b7796d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:06.604172 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.604144 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:45:06.605616 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.605593 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:45:06.715599 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.715557 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-c6c5d7"] Sep 19 12:45:06.718802 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.718772 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-c6c5d7"] Sep 19 12:45:06.829035 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.828998 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-576e23"] Sep 19 12:45:06.829636 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.829614 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" containerName="test-curl-7564-c6c5d7" Sep 19 12:45:06.829778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.829637 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" containerName="test-curl-7564-c6c5d7" Sep 19 12:45:06.829843 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.829820 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" containerName="test-curl-7564-c6c5d7" Sep 19 12:45:06.834709 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.834654 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:06.837634 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.837609 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-576e23"] Sep 19 12:45:06.924524 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:06.924483 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjx6x\" (UniqueName: \"kubernetes.io/projected/83546f3e-162d-40fa-9ee5-3aab7ef3a280-kube-api-access-wjx6x\") pod \"test-curl-7564-576e23\" (UID: \"83546f3e-162d-40fa-9ee5-3aab7ef3a280\") " pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:07.026077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:07.026033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjx6x\" (UniqueName: \"kubernetes.io/projected/83546f3e-162d-40fa-9ee5-3aab7ef3a280-kube-api-access-wjx6x\") pod \"test-curl-7564-576e23\" (UID: \"83546f3e-162d-40fa-9ee5-3aab7ef3a280\") " pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:07.035416 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:07.035395 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjx6x\" (UniqueName: \"kubernetes.io/projected/83546f3e-162d-40fa-9ee5-3aab7ef3a280-kube-api-access-wjx6x\") pod \"test-curl-7564-576e23\" (UID: \"83546f3e-162d-40fa-9ee5-3aab7ef3a280\") " pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:07.114878 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:07.114796 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-c6c5d7" container="test-curl-7564-c6c5d7" Sep 19 12:45:07.146535 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:07.146499 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:07.427590 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:07.427558 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-576e23"] Sep 19 12:45:07.428228 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:45:07.428205 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83546f3e_162d_40fa_9ee5_3aab7ef3a280.slice/crio-d8c79b2ec243c0ff01e0154df29e7ecbe74adb440398e6aad78a0e7450efa893 WatchSource:0}: Error finding container d8c79b2ec243c0ff01e0154df29e7ecbe74adb440398e6aad78a0e7450efa893: Status 404 returned error can't find the container with id d8c79b2ec243c0ff01e0154df29e7ecbe74adb440398e6aad78a0e7450efa893 Sep 19 12:45:08.124733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.124670 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-576e23" event={"ID":"83546f3e-162d-40fa-9ee5-3aab7ef3a280","Type":"ContainerStarted","Data":"1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8"} Sep 19 12:45:08.124733 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.124736 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-576e23" event={"ID":"83546f3e-162d-40fa-9ee5-3aab7ef3a280","Type":"ContainerStarted","Data":"d8c79b2ec243c0ff01e0154df29e7ecbe74adb440398e6aad78a0e7450efa893"} Sep 19 12:45:08.125216 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.124778 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-c6c5d7" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" containerName="test-curl-7564-c6c5d7" containerID="cri-o://c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029" gracePeriod=2 Sep 19 12:45:08.127233 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:08.127207 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-c6c5d7\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:08.141727 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.141641 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-576e23" podStartSLOduration=2.141621069 podStartE2EDuration="2.141621069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:45:08.140997218 +0000 UTC m=+1203.747847787" watchObservedRunningTime="2026-09-19 12:45:08.141621069 +0000 UTC m=+1203.748471649" Sep 19 12:45:08.143248 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:08.143221 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-c6c5d7\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:08.299033 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.299004 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:08.301261 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:08.301227 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-c6c5d7\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:08.338316 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.338286 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvz76\" (UniqueName: \"kubernetes.io/projected/0b8cf978-733c-42b4-b25a-21b0d9b99299-kube-api-access-xvz76\") pod \"0b8cf978-733c-42b4-b25a-21b0d9b99299\" (UID: \"0b8cf978-733c-42b4-b25a-21b0d9b99299\") " Sep 19 12:45:08.340560 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.340522 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8cf978-733c-42b4-b25a-21b0d9b99299-kube-api-access-xvz76" (OuterVolumeSpecName: "kube-api-access-xvz76") pod "0b8cf978-733c-42b4-b25a-21b0d9b99299" (UID: "0b8cf978-733c-42b4-b25a-21b0d9b99299"). InnerVolumeSpecName "kube-api-access-xvz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:45:08.439668 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.439567 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvz76\" (UniqueName: \"kubernetes.io/projected/0b8cf978-733c-42b4-b25a-21b0d9b99299-kube-api-access-xvz76\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:45:08.581097 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.581064 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8cf978-733c-42b4-b25a-21b0d9b99299" path="/var/lib/kubelet/pods/0b8cf978-733c-42b4-b25a-21b0d9b99299/volumes" Sep 19 12:45:08.731171 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.731081 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-576e23"] Sep 19 12:45:08.736552 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.736522 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-576e23"] Sep 19 12:45:08.847304 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.847263 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-a82d03"] Sep 19 12:45:08.847839 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.847825 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" containerName="test-curl-7564-576e23" Sep 19 12:45:08.847891 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.847842 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" containerName="test-curl-7564-576e23" Sep 19 12:45:08.847941 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.847932 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" containerName="test-curl-7564-576e23" Sep 19 12:45:08.860882 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.860853 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:08.862919 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.862888 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-a82d03"] Sep 19 12:45:08.944430 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:08.944393 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvg66\" (UniqueName: \"kubernetes.io/projected/feb901d8-f76b-4920-b7fb-c40496cbd4d3-kube-api-access-kvg66\") pod \"test-curl-7564-a82d03\" (UID: \"feb901d8-f76b-4920-b7fb-c40496cbd4d3\") " pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:09.045930 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.045896 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvg66\" (UniqueName: \"kubernetes.io/projected/feb901d8-f76b-4920-b7fb-c40496cbd4d3-kube-api-access-kvg66\") pod \"test-curl-7564-a82d03\" (UID: \"feb901d8-f76b-4920-b7fb-c40496cbd4d3\") " pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:09.054852 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.054820 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvg66\" (UniqueName: \"kubernetes.io/projected/feb901d8-f76b-4920-b7fb-c40496cbd4d3-kube-api-access-kvg66\") pod \"test-curl-7564-a82d03\" (UID: \"feb901d8-f76b-4920-b7fb-c40496cbd4d3\") " pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:09.132579 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.132538 2585 generic.go:356] "Generic (PLEG): container finished" podID="0b8cf978-733c-42b4-b25a-21b0d9b99299" containerID="adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e" exitCode=137 Sep 19 12:45:09.133045 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.132602 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-b7796d" Sep 19 12:45:09.133045 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.132639 2585 scope.go:117] "RemoveContainer" containerID="adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e" Sep 19 12:45:09.133045 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:09.132862 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-576e23" container="test-curl-7564-576e23" Sep 19 12:45:09.142494 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.142476 2585 scope.go:117] "RemoveContainer" containerID="adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e" Sep 19 12:45:09.142844 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:09.142826 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e\": container with ID starting with adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e not found: ID does not exist" containerID="adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e" Sep 19 12:45:09.142914 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.142857 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e"} err="failed to get container status \"adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e\": rpc error: code = NotFound desc = could not find container \"adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e\": container with ID starting with adcaa9fd9a64d1369bacbdd6d9cab073d4283465d6e2942ecc7d171e7551382e not found: ID does not exist" Sep 19 12:45:09.171310 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.171267 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:09.448519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:09.448483 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-a82d03"] Sep 19 12:45:09.449781 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:45:09.449755 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeb901d8_f76b_4920_b7fb_c40496cbd4d3.slice/crio-3eb91741169001cacca8418498ab34e9783ef93968688e5b8fdcdb0ef9424801 WatchSource:0}: Error finding container 3eb91741169001cacca8418498ab34e9783ef93968688e5b8fdcdb0ef9424801: Status 404 returned error can't find the container with id 3eb91741169001cacca8418498ab34e9783ef93968688e5b8fdcdb0ef9424801 Sep 19 12:45:10.142987 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.142957 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-a82d03" event={"ID":"feb901d8-f76b-4920-b7fb-c40496cbd4d3","Type":"ContainerStarted","Data":"f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1"} Sep 19 12:45:10.143383 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.143006 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-a82d03" event={"ID":"feb901d8-f76b-4920-b7fb-c40496cbd4d3","Type":"ContainerStarted","Data":"3eb91741169001cacca8418498ab34e9783ef93968688e5b8fdcdb0ef9424801"} Sep 19 12:45:10.144156 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.144132 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-576e23" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" containerName="test-curl-7564-576e23" containerID="cri-o://1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8" gracePeriod=2 Sep 19 12:45:10.160123 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:10.160077 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-576e23\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:10.160571 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.160532 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7564-a82d03" podStartSLOduration=2.160517333 podStartE2EDuration="2.160517333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:45:10.157756522 +0000 UTC m=+1205.764607113" watchObservedRunningTime="2026-09-19 12:45:10.160517333 +0000 UTC m=+1205.767367911" Sep 19 12:45:10.332755 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.332724 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:10.334931 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:10.334905 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-576e23\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:10.362484 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.362445 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m89l\" (UniqueName: \"kubernetes.io/projected/478c59e6-fd7a-4730-8161-86a770f3f6b1-kube-api-access-5m89l\") pod \"478c59e6-fd7a-4730-8161-86a770f3f6b1\" (UID: \"478c59e6-fd7a-4730-8161-86a770f3f6b1\") " Sep 19 12:45:10.364806 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.364779 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478c59e6-fd7a-4730-8161-86a770f3f6b1-kube-api-access-5m89l" (OuterVolumeSpecName: "kube-api-access-5m89l") pod "478c59e6-fd7a-4730-8161-86a770f3f6b1" (UID: "478c59e6-fd7a-4730-8161-86a770f3f6b1"). InnerVolumeSpecName "kube-api-access-5m89l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:45:10.464399 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.464314 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m89l\" (UniqueName: \"kubernetes.io/projected/478c59e6-fd7a-4730-8161-86a770f3f6b1-kube-api-access-5m89l\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:45:10.582220 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.582185 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478c59e6-fd7a-4730-8161-86a770f3f6b1" path="/var/lib/kubelet/pods/478c59e6-fd7a-4730-8161-86a770f3f6b1/volumes" Sep 19 12:45:10.822474 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.822434 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-a82d03"] Sep 19 12:45:10.827278 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:10.827242 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-a82d03"] Sep 19 12:45:11.152148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:11.152045 2585 generic.go:356] "Generic (PLEG): container finished" podID="478c59e6-fd7a-4730-8161-86a770f3f6b1" containerID="c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029" exitCode=137 Sep 19 12:45:11.152148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:11.152102 2585 scope.go:117] "RemoveContainer" containerID="c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029" Sep 19 12:45:11.152148 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:11.152111 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-c6c5d7" Sep 19 12:45:11.152734 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:11.152391 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7564-a82d03" container="test-curl-7564-a82d03" Sep 19 12:45:11.162701 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:11.162666 2585 scope.go:117] "RemoveContainer" containerID="c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029" Sep 19 12:45:11.163023 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:11.163002 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029\": container with ID starting with c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029 not found: ID does not exist" containerID="c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029" Sep 19 12:45:11.163092 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:11.163032 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029"} err="failed to get container status \"c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029\": rpc error: code = NotFound desc = could not find container \"c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029\": container with ID starting with c8b008acb9bf434fc122f95713d989f2f90f00fa131d292858510ba2af260029 not found: ID does not exist" Sep 19 12:45:12.162940 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:12.162889 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7564-a82d03" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" containerName="test-curl-7564-a82d03" containerID="cri-o://f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1" gracePeriod=2 Sep 19 12:45:12.165202 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:12.165174 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-a82d03\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:12.349815 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:12.349780 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:12.351813 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:12.351786 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-a82d03\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:12.386956 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:12.386918 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjx6x\" (UniqueName: \"kubernetes.io/projected/83546f3e-162d-40fa-9ee5-3aab7ef3a280-kube-api-access-wjx6x\") pod \"83546f3e-162d-40fa-9ee5-3aab7ef3a280\" (UID: \"83546f3e-162d-40fa-9ee5-3aab7ef3a280\") " Sep 19 12:45:12.389212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:12.389187 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83546f3e-162d-40fa-9ee5-3aab7ef3a280-kube-api-access-wjx6x" (OuterVolumeSpecName: "kube-api-access-wjx6x") pod "83546f3e-162d-40fa-9ee5-3aab7ef3a280" (UID: "83546f3e-162d-40fa-9ee5-3aab7ef3a280"). InnerVolumeSpecName "kube-api-access-wjx6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:45:12.488202 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:12.488109 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjx6x\" (UniqueName: \"kubernetes.io/projected/83546f3e-162d-40fa-9ee5-3aab7ef3a280-kube-api-access-wjx6x\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:45:12.580872 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:12.580839 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" path="/var/lib/kubelet/pods/83546f3e-162d-40fa-9ee5-3aab7ef3a280/volumes" Sep 19 12:45:13.172255 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:13.172216 2585 generic.go:356] "Generic (PLEG): container finished" podID="83546f3e-162d-40fa-9ee5-3aab7ef3a280" containerID="1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8" exitCode=137 Sep 19 12:45:13.172675 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:13.172291 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-576e23" Sep 19 12:45:13.172675 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:13.172331 2585 scope.go:117] "RemoveContainer" containerID="1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8" Sep 19 12:45:13.183168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:13.183147 2585 scope.go:117] "RemoveContainer" containerID="1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8" Sep 19 12:45:13.183511 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:13.183490 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8\": container with ID starting with 1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8 not found: ID does not exist" containerID="1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8" Sep 19 12:45:13.183576 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:13.183526 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8"} err="failed to get container status \"1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8\": rpc error: code = NotFound desc = could not find container \"1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8\": container with ID starting with 1f33cc6d93732df30541d95ecab0c5ee035ec3eb91fbc46a11acaed71b29a8f8 not found: ID does not exist" Sep 19 12:45:14.354700 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:14.354655 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:14.411061 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:14.411029 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvg66\" (UniqueName: \"kubernetes.io/projected/feb901d8-f76b-4920-b7fb-c40496cbd4d3-kube-api-access-kvg66\") pod \"feb901d8-f76b-4920-b7fb-c40496cbd4d3\" (UID: \"feb901d8-f76b-4920-b7fb-c40496cbd4d3\") " Sep 19 12:45:14.413363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:14.413334 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb901d8-f76b-4920-b7fb-c40496cbd4d3-kube-api-access-kvg66" (OuterVolumeSpecName: "kube-api-access-kvg66") pod "feb901d8-f76b-4920-b7fb-c40496cbd4d3" (UID: "feb901d8-f76b-4920-b7fb-c40496cbd4d3"). InnerVolumeSpecName "kube-api-access-kvg66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:45:14.513009 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:14.512969 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvg66\" (UniqueName: \"kubernetes.io/projected/feb901d8-f76b-4920-b7fb-c40496cbd4d3-kube-api-access-kvg66\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:45:14.581128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:14.581091 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" path="/var/lib/kubelet/pods/feb901d8-f76b-4920-b7fb-c40496cbd4d3/volumes" Sep 19 12:45:15.191614 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:15.191574 2585 generic.go:356] "Generic (PLEG): container finished" podID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" containerID="f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1" exitCode=137 Sep 19 12:45:15.191800 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:15.191619 2585 scope.go:117] "RemoveContainer" containerID="f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1" Sep 19 12:45:15.191800 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:15.191620 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-a82d03" Sep 19 12:45:15.202187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:15.201997 2585 scope.go:117] "RemoveContainer" containerID="f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1" Sep 19 12:45:15.202383 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:15.202359 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1\": container with ID starting with f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1 not found: ID does not exist" containerID="f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1" Sep 19 12:45:15.202431 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:15.202395 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1"} err="failed to get container status \"f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1\": rpc error: code = NotFound desc = could not find container \"f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1\": container with ID starting with f1b0f80a7f0eae8521a9b2a1933a2041413fe8e8cc846985fbdc8f463efc4cd1 not found: ID does not exist" Sep 19 12:45:19.877725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.877662 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl"] Sep 19 12:45:19.879211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.879178 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" containerName="test-curl-7564-a82d03" Sep 19 12:45:19.879211 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.879213 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" containerName="test-curl-7564-a82d03" Sep 19 12:45:19.879399 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.879364 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb901d8-f76b-4920-b7fb-c40496cbd4d3" containerName="test-curl-7564-a82d03" Sep 19 12:45:19.885025 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.884989 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl"] Sep 19 12:45:19.885174 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.885114 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:19.887534 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.887510 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e9a75910-serving-cert\"" Sep 19 12:45:19.973372 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.973318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b123fea1-b0af-48a8-9b7e-b99aac6efccb-openshift-service-ca\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:19.973588 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.973489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-metrics-certs\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:19.973706 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.973631 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmbd\" (UniqueName: \"kubernetes.io/projected/b123fea1-b0af-48a8-9b7e-b99aac6efccb-kube-api-access-xjmbd\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:19.973706 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:19.973677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.074513 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.074472 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-metrics-certs\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.074754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.074547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmbd\" (UniqueName: \"kubernetes.io/projected/b123fea1-b0af-48a8-9b7e-b99aac6efccb-kube-api-access-xjmbd\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.074754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.074568 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.074754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.074668 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b123fea1-b0af-48a8-9b7e-b99aac6efccb-openshift-service-ca\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.075025 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:20.074996 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-serving-cert: secret "maas-api-e2e-mt-e9a75910-serving-cert" not found Sep 19 12:45:20.075108 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:20.075092 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls podName:b123fea1-b0af-48a8-9b7e-b99aac6efccb nodeName:}" failed. No retries permitted until 2026-09-19 12:45:20.575053656 +0000 UTC m=+1216.181904232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls") pod "maas-api-e2e-mt-e9a75910-566f79d9-2btfl" (UID: "b123fea1-b0af-48a8-9b7e-b99aac6efccb") : secret "maas-api-e2e-mt-e9a75910-serving-cert" not found Sep 19 12:45:20.075479 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.075426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b123fea1-b0af-48a8-9b7e-b99aac6efccb-openshift-service-ca\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.077561 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.077533 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-metrics-certs\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.083953 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.083919 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmbd\" (UniqueName: \"kubernetes.io/projected/b123fea1-b0af-48a8-9b7e-b99aac6efccb-kube-api-access-xjmbd\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.580975 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:20.580932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:20.581170 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:20.581063 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-serving-cert: secret "maas-api-e2e-mt-e9a75910-serving-cert" not found Sep 19 12:45:20.581170 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:45:20.581131 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls podName:b123fea1-b0af-48a8-9b7e-b99aac6efccb nodeName:}" failed. No retries permitted until 2026-09-19 12:45:21.581116287 +0000 UTC m=+1217.187966844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls") pod "maas-api-e2e-mt-e9a75910-566f79d9-2btfl" (UID: "b123fea1-b0af-48a8-9b7e-b99aac6efccb") : secret "maas-api-e2e-mt-e9a75910-serving-cert" not found Sep 19 12:45:21.593902 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:21.593850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:21.596578 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:21.596554 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") pod \"maas-api-e2e-mt-e9a75910-566f79d9-2btfl\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:21.698916 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:21.698866 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:21.979151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:21.979116 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl"] Sep 19 12:45:21.981114 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:45:21.981087 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb123fea1_b0af_48a8_9b7e_b99aac6efccb.slice/crio-0f87a7e1552fca8a4d2479fac138a9bcfb0534b98d6d0685ed905051468722b9 WatchSource:0}: Error finding container 0f87a7e1552fca8a4d2479fac138a9bcfb0534b98d6d0685ed905051468722b9: Status 404 returned error can't find the container with id 0f87a7e1552fca8a4d2479fac138a9bcfb0534b98d6d0685ed905051468722b9 Sep 19 12:45:22.242794 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:22.242673 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" event={"ID":"b123fea1-b0af-48a8-9b7e-b99aac6efccb","Type":"ContainerStarted","Data":"0f87a7e1552fca8a4d2479fac138a9bcfb0534b98d6d0685ed905051468722b9"} Sep 19 12:45:23.251631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:23.251592 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" event={"ID":"b123fea1-b0af-48a8-9b7e-b99aac6efccb","Type":"ContainerStarted","Data":"a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f"} Sep 19 12:45:23.252084 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:23.251726 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:45:23.273449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:23.273387 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" podStartSLOduration=3.815346575 podStartE2EDuration="4.273367274s" totalImagesPullingTime="458.020699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:45:23.270299642 +0000 UTC m=+1218.877150262" watchObservedRunningTime="2026-09-19 12:45:23.273367274 +0000 UTC m=+1218.880217852" Sep 19 12:45:29.264421 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:45:29.264389 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:46:18.639276 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:18.639236 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9"] Sep 19 12:46:18.657629 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:18.657579 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" podUID="10ab80c1-7e37-4157-8732-ee49da02ef3f" containerName="maas-api" containerID="cri-o://a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc" gracePeriod=30 Sep 19 12:46:18.945571 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:18.945534 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:46:19.030354 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.030318 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-metrics-certs\") pod \"10ab80c1-7e37-4157-8732-ee49da02ef3f\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " Sep 19 12:46:19.030354 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.030352 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drjps\" (UniqueName: \"kubernetes.io/projected/10ab80c1-7e37-4157-8732-ee49da02ef3f-kube-api-access-drjps\") pod \"10ab80c1-7e37-4157-8732-ee49da02ef3f\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " Sep 19 12:46:19.030617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.030389 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls\") pod \"10ab80c1-7e37-4157-8732-ee49da02ef3f\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " Sep 19 12:46:19.030617 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.030449 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab80c1-7e37-4157-8732-ee49da02ef3f-openshift-service-ca\") pod \"10ab80c1-7e37-4157-8732-ee49da02ef3f\" (UID: \"10ab80c1-7e37-4157-8732-ee49da02ef3f\") " Sep 19 12:46:19.030894 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.030870 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ab80c1-7e37-4157-8732-ee49da02ef3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10ab80c1-7e37-4157-8732-ee49da02ef3f" (UID: "10ab80c1-7e37-4157-8732-ee49da02ef3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:46:19.032698 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.032662 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10ab80c1-7e37-4157-8732-ee49da02ef3f" (UID: "10ab80c1-7e37-4157-8732-ee49da02ef3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:46:19.033174 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.033151 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "10ab80c1-7e37-4157-8732-ee49da02ef3f" (UID: "10ab80c1-7e37-4157-8732-ee49da02ef3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:46:19.033228 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.033174 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ab80c1-7e37-4157-8732-ee49da02ef3f-kube-api-access-drjps" (OuterVolumeSpecName: "kube-api-access-drjps") pod "10ab80c1-7e37-4157-8732-ee49da02ef3f" (UID: "10ab80c1-7e37-4157-8732-ee49da02ef3f"). InnerVolumeSpecName "kube-api-access-drjps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:46:19.132189 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.132152 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:19.132189 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.132185 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjps\" (UniqueName: \"kubernetes.io/projected/10ab80c1-7e37-4157-8732-ee49da02ef3f-kube-api-access-drjps\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:19.132189 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.132197 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ab80c1-7e37-4157-8732-ee49da02ef3f-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:19.132465 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.132205 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab80c1-7e37-4157-8732-ee49da02ef3f-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:19.665927 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.665889 2585 generic.go:356] "Generic (PLEG): container finished" podID="10ab80c1-7e37-4157-8732-ee49da02ef3f" containerID="a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc" exitCode=0 Sep 19 12:46:19.666378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.665935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" event={"ID":"10ab80c1-7e37-4157-8732-ee49da02ef3f","Type":"ContainerDied","Data":"a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc"} Sep 19 12:46:19.666378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.665973 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" event={"ID":"10ab80c1-7e37-4157-8732-ee49da02ef3f","Type":"ContainerDied","Data":"ed0e81ebd3bce8c88403a73dcf4d6542735ef13550279152e473f1bdd3ad1dcf"} Sep 19 12:46:19.666378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.665937 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9" Sep 19 12:46:19.666378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.666021 2585 scope.go:117] "RemoveContainer" containerID="a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc" Sep 19 12:46:19.678293 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.678274 2585 scope.go:117] "RemoveContainer" containerID="a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc" Sep 19 12:46:19.678668 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:46:19.678646 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc\": container with ID starting with a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc not found: ID does not exist" containerID="a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc" Sep 19 12:46:19.678748 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.678677 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc"} err="failed to get container status \"a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc\": rpc error: code = NotFound desc = could not find container \"a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc\": container with ID starting with a0945af67485ee8eec0d67e03ba337f1b7ac5e181f18eac8a7a0335bcdf0d5cc not found: ID does not exist" Sep 19 12:46:19.696969 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.696924 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9"] Sep 19 12:46:19.699883 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:19.699858 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61583b-78d57fbf85-sh8d9"] Sep 19 12:46:20.582410 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:20.582378 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ab80c1-7e37-4157-8732-ee49da02ef3f" path="/var/lib/kubelet/pods/10ab80c1-7e37-4157-8732-ee49da02ef3f/volumes" Sep 19 12:46:45.365832 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:45.365796 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt"] Sep 19 12:46:45.366283 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:45.366050 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" containerID="cri-o://3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906" gracePeriod=30 Sep 19 12:46:50.643195 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.643140 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:46:50.771731 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771663 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-certs\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.771955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771753 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-socket\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.771955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771801 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-credential-socket\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.771955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771862 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t4bz\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-kube-api-access-6t4bz\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.771955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771910 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-token\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.771955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771950 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-data\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.772266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771971 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istiod-ca-cert\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.772266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.771954 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:46:50.772266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772016 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:46:50.772266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772044 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-envoy\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.772266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772110 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-podinfo\") pod \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\" (UID: \"1ff40a8f-0654-4c20-afd6-95a27fb740e7\") " Sep 19 12:46:50.772266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772114 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:46:50.772583 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772387 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:46:50.772583 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772549 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.772583 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772569 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-workload-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.772771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772584 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-credential-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.772771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772586 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:46:50.772771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.772598 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istiod-ca-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.774575 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.774539 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:46:50.774778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.774754 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-kube-api-access-6t4bz" (OuterVolumeSpecName: "kube-api-access-6t4bz") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "kube-api-access-6t4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:46:50.775001 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.774979 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:46:50.775260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.775228 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ff40a8f-0654-4c20-afd6-95a27fb740e7" (UID: "1ff40a8f-0654-4c20-afd6-95a27fb740e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 12:46:50.873165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.873140 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4bz\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-kube-api-access-6t4bz\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.873165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.873164 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-token\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.873379 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.873174 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-data\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.873379 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.873184 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-envoy\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.873379 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.873192 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff40a8f-0654-4c20-afd6-95a27fb740e7-istio-podinfo\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:46:50.888748 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.888720 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerID="3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906" exitCode=0 Sep 19 12:46:50.888867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.888757 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" Sep 19 12:46:50.888867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.888795 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" event={"ID":"1ff40a8f-0654-4c20-afd6-95a27fb740e7","Type":"ContainerDied","Data":"3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906"} Sep 19 12:46:50.888867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.888847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt" event={"ID":"1ff40a8f-0654-4c20-afd6-95a27fb740e7","Type":"ContainerDied","Data":"a461b7224fca4b67e049b8f033114ca6b1c679a5649709cc1d2b2740baefd2d8"} Sep 19 12:46:50.888867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.888865 2585 scope.go:117] "RemoveContainer" containerID="3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906" Sep 19 12:46:50.899946 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.899931 2585 scope.go:117] "RemoveContainer" containerID="f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973" Sep 19 12:46:50.910583 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.910567 2585 scope.go:117] "RemoveContainer" containerID="3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906" Sep 19 12:46:50.910888 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:46:50.910870 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906\": container with ID starting with 3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906 not found: ID does not exist" containerID="3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906" Sep 19 12:46:50.910954 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.910897 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906"} err="failed to get container status \"3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906\": rpc error: code = NotFound desc = could not find container \"3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906\": container with ID starting with 3c10392d17d0c6d6ec864774c731d00dd8e4ef527cb2bb7145a14a30aa2a6906 not found: ID does not exist" Sep 19 12:46:50.910954 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.910913 2585 scope.go:117] "RemoveContainer" containerID="f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973" Sep 19 12:46:50.911200 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:46:50.911178 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973\": container with ID starting with f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973 not found: ID does not exist" containerID="f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973" Sep 19 12:46:50.911248 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.911209 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973"} err="failed to get container status \"f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973\": rpc error: code = NotFound desc = could not find container \"f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973\": container with ID starting with f29f5a5ec4c5b1e3d3586ae8a1a62647fb94b414ed3a60254baae4407a9c2973 not found: ID does not exist" Sep 19 12:46:50.912536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.912502 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt"] Sep 19 12:46:50.915947 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:50.915930 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5be349-openshift-default-bcc9d67b-2dmmt"] Sep 19 12:46:52.588868 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.588812 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" path="/var/lib/kubelet/pods/1ff40a8f-0654-4c20-afd6-95a27fb740e7/volumes" Sep 19 12:46:52.757128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.757093 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc"] Sep 19 12:46:52.757967 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.757947 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ab80c1-7e37-4157-8732-ee49da02ef3f" containerName="maas-api" Sep 19 12:46:52.758033 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.757992 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ab80c1-7e37-4157-8732-ee49da02ef3f" containerName="maas-api" Sep 19 12:46:52.758033 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758026 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" Sep 19 12:46:52.758102 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758036 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" Sep 19 12:46:52.758102 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758063 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" Sep 19 12:46:52.758102 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758072 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" Sep 19 12:46:52.758218 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758206 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ab80c1-7e37-4157-8732-ee49da02ef3f" containerName="maas-api" Sep 19 12:46:52.758256 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758224 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" Sep 19 12:46:52.758256 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.758237 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff40a8f-0654-4c20-afd6-95a27fb740e7" containerName="istio-proxy" Sep 19 12:46:52.762631 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.762603 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.765122 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.765096 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b0e30e-serving-cert\"" Sep 19 12:46:52.767674 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.767650 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc"] Sep 19 12:46:52.898601 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.898509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.898795 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.898609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.898795 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.898654 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.898795 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.898676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prw2c\" (UniqueName: \"kubernetes.io/projected/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-kube-api-access-prw2c\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.999266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.999236 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.999456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.999300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.999456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.999340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.999456 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:52.999366 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prw2c\" (UniqueName: \"kubernetes.io/projected/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-kube-api-access-prw2c\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:52.999456 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:46:52.999421 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-serving-cert: secret "maas-api-e2e-shared-a-w3-b0e30e-serving-cert" not found Sep 19 12:46:52.999717 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:46:52.999488 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls podName:0bbf5a75-b706-4958-a6c5-83aa91a96ed7 nodeName:}" failed. No retries permitted until 2026-09-19 12:46:53.499466826 +0000 UTC m=+1309.106317396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls") pod "maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" (UID: "0bbf5a75-b706-4958-a6c5-83aa91a96ed7") : secret "maas-api-e2e-shared-a-w3-b0e30e-serving-cert" not found Sep 19 12:46:53.000035 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.000000 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:53.002083 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.002059 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:53.009029 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.009007 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prw2c\" (UniqueName: \"kubernetes.io/projected/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-kube-api-access-prw2c\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:53.504753 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.504710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:53.507287 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.507263 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:53.677979 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.677951 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:53.966090 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:53.966054 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc"] Sep 19 12:46:53.967415 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:46:53.967391 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbf5a75_b706_4958_a6c5_83aa91a96ed7.slice/crio-ae26a42bf390a6468b751a5e3b2c5c8b3c08c983c3c57fa65df8ef868e4ab116 WatchSource:0}: Error finding container ae26a42bf390a6468b751a5e3b2c5c8b3c08c983c3c57fa65df8ef868e4ab116: Status 404 returned error can't find the container with id ae26a42bf390a6468b751a5e3b2c5c8b3c08c983c3c57fa65df8ef868e4ab116 Sep 19 12:46:54.920282 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:54.920245 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" event={"ID":"0bbf5a75-b706-4958-a6c5-83aa91a96ed7","Type":"ContainerStarted","Data":"fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8"} Sep 19 12:46:54.920712 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:54.920403 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:46:54.920712 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:54.920422 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" event={"ID":"0bbf5a75-b706-4958-a6c5-83aa91a96ed7","Type":"ContainerStarted","Data":"ae26a42bf390a6468b751a5e3b2c5c8b3c08c983c3c57fa65df8ef868e4ab116"} Sep 19 12:46:54.939961 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:46:54.939911 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" podStartSLOduration=2.518688216 podStartE2EDuration="2.939896539s" totalImagesPullingTime="421.208323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:46:54.936169497 +0000 UTC m=+1310.543020106" watchObservedRunningTime="2026-09-19 12:46:54.939896539 +0000 UTC m=+1310.546747118" Sep 19 12:47:00.932920 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:00.932889 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 12:47:26.768783 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.768747 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz"] Sep 19 12:47:26.774442 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.774416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.777248 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.777223 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5d6124-serving-cert\"" Sep 19 12:47:26.783498 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.783330 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz"] Sep 19 12:47:26.838363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.838332 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.838479 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.838374 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l28v\" (UniqueName: \"kubernetes.io/projected/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-kube-api-access-8l28v\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.838479 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.838443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.838554 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.838509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.939109 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.939068 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.939285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.939125 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l28v\" (UniqueName: \"kubernetes.io/projected/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-kube-api-access-8l28v\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.939285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.939205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.939416 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.939311 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.939416 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:47:26.939358 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-serving-cert: secret "maas-api-e2e-shared-b-w3-5d6124-serving-cert" not found Sep 19 12:47:26.939537 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:47:26.939428 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls podName:472b95cb-4cbd-4bc0-84a3-cb17e47c1d75 nodeName:}" failed. No retries permitted until 2026-09-19 12:47:27.439408228 +0000 UTC m=+1343.046258797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" (UID: "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75") : secret "maas-api-e2e-shared-b-w3-5d6124-serving-cert" not found Sep 19 12:47:26.939884 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.939859 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.942272 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.942241 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:26.958749 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:26.958730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l28v\" (UniqueName: \"kubernetes.io/projected/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-kube-api-access-8l28v\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:27.445245 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:27.445210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:27.447790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:27.447771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:27.689800 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:27.689767 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:27.967220 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:27.967193 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz"] Sep 19 12:47:27.969782 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:47:27.969742 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472b95cb_4cbd_4bc0_84a3_cb17e47c1d75.slice/crio-ea96e9319b854ca4fb7c3162a581b69d260b51b29192d95d3fd09bb859503161 WatchSource:0}: Error finding container ea96e9319b854ca4fb7c3162a581b69d260b51b29192d95d3fd09bb859503161: Status 404 returned error can't find the container with id ea96e9319b854ca4fb7c3162a581b69d260b51b29192d95d3fd09bb859503161 Sep 19 12:47:28.148865 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:28.148826 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" event={"ID":"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75","Type":"ContainerStarted","Data":"ea96e9319b854ca4fb7c3162a581b69d260b51b29192d95d3fd09bb859503161"} Sep 19 12:47:29.157771 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:29.157733 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" event={"ID":"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75","Type":"ContainerStarted","Data":"98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef"} Sep 19 12:47:29.158160 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:29.157837 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:47:29.177033 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:29.176974 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" podStartSLOduration=2.7398422829999998 podStartE2EDuration="3.176960894s" totalImagesPullingTime="437.118611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:47:29.175097974 +0000 UTC m=+1344.781948574" watchObservedRunningTime="2026-09-19 12:47:29.176960894 +0000 UTC m=+1344.783811486" Sep 19 12:47:35.171128 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:47:35.171099 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 12:48:23.288393 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:23.288352 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx"] Sep 19 12:48:26.845108 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:26.845077 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl"] Sep 19 12:48:26.845518 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:26.845270 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" podUID="b123fea1-b0af-48a8-9b7e-b99aac6efccb" containerName="maas-api" containerID="cri-o://a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f" gracePeriod=30 Sep 19 12:48:27.136233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.136210 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:48:27.310955 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.310915 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjmbd\" (UniqueName: \"kubernetes.io/projected/b123fea1-b0af-48a8-9b7e-b99aac6efccb-kube-api-access-xjmbd\") pod \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " Sep 19 12:48:27.311116 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.311038 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") pod \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " Sep 19 12:48:27.311116 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.311075 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b123fea1-b0af-48a8-9b7e-b99aac6efccb-openshift-service-ca\") pod \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " Sep 19 12:48:27.311116 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.311106 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-metrics-certs\") pod \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\" (UID: \"b123fea1-b0af-48a8-9b7e-b99aac6efccb\") " Sep 19 12:48:27.311429 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.311402 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b123fea1-b0af-48a8-9b7e-b99aac6efccb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b123fea1-b0af-48a8-9b7e-b99aac6efccb" (UID: "b123fea1-b0af-48a8-9b7e-b99aac6efccb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:48:27.311604 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.311581 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b123fea1-b0af-48a8-9b7e-b99aac6efccb-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:27.313542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.313514 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b123fea1-b0af-48a8-9b7e-b99aac6efccb" (UID: "b123fea1-b0af-48a8-9b7e-b99aac6efccb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:48:27.313874 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.313851 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b123fea1-b0af-48a8-9b7e-b99aac6efccb" (UID: "b123fea1-b0af-48a8-9b7e-b99aac6efccb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:48:27.313987 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.313894 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b123fea1-b0af-48a8-9b7e-b99aac6efccb-kube-api-access-xjmbd" (OuterVolumeSpecName: "kube-api-access-xjmbd") pod "b123fea1-b0af-48a8-9b7e-b99aac6efccb" (UID: "b123fea1-b0af-48a8-9b7e-b99aac6efccb"). InnerVolumeSpecName "kube-api-access-xjmbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:48:27.412741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.412654 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjmbd\" (UniqueName: \"kubernetes.io/projected/b123fea1-b0af-48a8-9b7e-b99aac6efccb-kube-api-access-xjmbd\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:27.412741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.412712 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:27.412741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.412729 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b123fea1-b0af-48a8-9b7e-b99aac6efccb-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:27.555778 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.555750 2585 generic.go:356] "Generic (PLEG): container finished" podID="b123fea1-b0af-48a8-9b7e-b99aac6efccb" containerID="a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f" exitCode=0 Sep 19 12:48:27.555949 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.555796 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" Sep 19 12:48:27.555949 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.555843 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" event={"ID":"b123fea1-b0af-48a8-9b7e-b99aac6efccb","Type":"ContainerDied","Data":"a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f"} Sep 19 12:48:27.555949 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.555892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl" event={"ID":"b123fea1-b0af-48a8-9b7e-b99aac6efccb","Type":"ContainerDied","Data":"0f87a7e1552fca8a4d2479fac138a9bcfb0534b98d6d0685ed905051468722b9"} Sep 19 12:48:27.555949 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.555920 2585 scope.go:117] "RemoveContainer" containerID="a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f" Sep 19 12:48:27.566728 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.566707 2585 scope.go:117] "RemoveContainer" containerID="a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f" Sep 19 12:48:27.566983 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:48:27.566965 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f\": container with ID starting with a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f not found: ID does not exist" containerID="a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f" Sep 19 12:48:27.567074 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.566992 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f"} err="failed to get container status \"a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f\": rpc error: code = NotFound desc = could not find container \"a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f\": container with ID starting with a89e04d9ac7442d38881f046047e3b4d77d23055a1b3fda577aed028faa35a1f not found: ID does not exist" Sep 19 12:48:27.580522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.580483 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl"] Sep 19 12:48:27.597945 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:27.597900 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9a75910-566f79d9-2btfl"] Sep 19 12:48:28.580741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:28.580700 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b123fea1-b0af-48a8-9b7e-b99aac6efccb" path="/var/lib/kubelet/pods/b123fea1-b0af-48a8-9b7e-b99aac6efccb/volumes" Sep 19 12:48:38.309735 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.309667 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" podUID="582de267-73ae-458f-8d27-34667310ffd8" containerName="main" containerID="cri-o://61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963" gracePeriod=45 Sep 19 12:48:38.586154 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.586082 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:48:38.623249 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623209 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-tmp-dir\") pod \"582de267-73ae-458f-8d27-34667310ffd8\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " Sep 19 12:48:38.623422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623275 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-model-cache\") pod \"582de267-73ae-458f-8d27-34667310ffd8\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " Sep 19 12:48:38.623422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623340 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9kg9\" (UniqueName: \"kubernetes.io/projected/582de267-73ae-458f-8d27-34667310ffd8-kube-api-access-f9kg9\") pod \"582de267-73ae-458f-8d27-34667310ffd8\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " Sep 19 12:48:38.623422 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623373 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-dshm\") pod \"582de267-73ae-458f-8d27-34667310ffd8\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " Sep 19 12:48:38.623613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623431 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-home\") pod \"582de267-73ae-458f-8d27-34667310ffd8\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " Sep 19 12:48:38.623613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623444 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "582de267-73ae-458f-8d27-34667310ffd8" (UID: "582de267-73ae-458f-8d27-34667310ffd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:48:38.623613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623459 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582de267-73ae-458f-8d27-34667310ffd8-tls-certs\") pod \"582de267-73ae-458f-8d27-34667310ffd8\" (UID: \"582de267-73ae-458f-8d27-34667310ffd8\") " Sep 19 12:48:38.623613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623565 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "582de267-73ae-458f-8d27-34667310ffd8" (UID: "582de267-73ae-458f-8d27-34667310ffd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:48:38.623613 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623587 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-home" (OuterVolumeSpecName: "home") pod "582de267-73ae-458f-8d27-34667310ffd8" (UID: "582de267-73ae-458f-8d27-34667310ffd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:48:38.623925 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623865 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-tmp-dir\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:38.623925 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623885 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-model-cache\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:38.623925 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.623896 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-home\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:38.625962 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.625938 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-dshm" (OuterVolumeSpecName: "dshm") pod "582de267-73ae-458f-8d27-34667310ffd8" (UID: "582de267-73ae-458f-8d27-34667310ffd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:48:38.626387 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.626364 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582de267-73ae-458f-8d27-34667310ffd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "582de267-73ae-458f-8d27-34667310ffd8" (UID: "582de267-73ae-458f-8d27-34667310ffd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:48:38.626469 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.626381 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582de267-73ae-458f-8d27-34667310ffd8-kube-api-access-f9kg9" (OuterVolumeSpecName: "kube-api-access-f9kg9") pod "582de267-73ae-458f-8d27-34667310ffd8" (UID: "582de267-73ae-458f-8d27-34667310ffd8"). InnerVolumeSpecName "kube-api-access-f9kg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:48:38.637669 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.637642 2585 generic.go:356] "Generic (PLEG): container finished" podID="582de267-73ae-458f-8d27-34667310ffd8" containerID="61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963" exitCode=0 Sep 19 12:48:38.637796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.637748 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" Sep 19 12:48:38.637796 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.637772 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" event={"ID":"582de267-73ae-458f-8d27-34667310ffd8","Type":"ContainerDied","Data":"61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963"} Sep 19 12:48:38.637924 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.637809 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx" event={"ID":"582de267-73ae-458f-8d27-34667310ffd8","Type":"ContainerDied","Data":"e0d776c6c8278814eb1c604626524d8e0731a6d6cc324ab7d39e26f75134c381"} Sep 19 12:48:38.637924 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.637831 2585 scope.go:117] "RemoveContainer" containerID="61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963" Sep 19 12:48:38.651287 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.651269 2585 scope.go:117] "RemoveContainer" containerID="61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963" Sep 19 12:48:38.651574 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:48:38.651555 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963\": container with ID starting with 61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963 not found: ID does not exist" containerID="61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963" Sep 19 12:48:38.651628 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.651582 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963"} err="failed to get container status \"61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963\": rpc error: code = NotFound desc = could not find container \"61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963\": container with ID starting with 61713dcedeeaeaabc153aeaf56f37bafa7ed3254d231a7dbc9810f2b4a2d8963 not found: ID does not exist" Sep 19 12:48:38.662616 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.662591 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx"] Sep 19 12:48:38.665426 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.665396 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/e2e-conflict-a-3ed226d2-kserve-7c449f86d9-rwhzx"] Sep 19 12:48:38.724735 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.724714 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582de267-73ae-458f-8d27-34667310ffd8-dshm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:38.724735 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.724733 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582de267-73ae-458f-8d27-34667310ffd8-tls-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:38.724860 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:38.724742 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kg9\" (UniqueName: \"kubernetes.io/projected/582de267-73ae-458f-8d27-34667310ffd8-kube-api-access-f9kg9\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:48:40.580084 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:48:40.580046 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582de267-73ae-458f-8d27-34667310ffd8" path="/var/lib/kubelet/pods/582de267-73ae-458f-8d27-34667310ffd8/volumes" Sep 19 12:49:02.624436 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.624406 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq"] Sep 19 12:49:02.626667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.625088 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b123fea1-b0af-48a8-9b7e-b99aac6efccb" containerName="maas-api" Sep 19 12:49:02.626667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.625108 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b123fea1-b0af-48a8-9b7e-b99aac6efccb" containerName="maas-api" Sep 19 12:49:02.626667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.625143 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582de267-73ae-458f-8d27-34667310ffd8" containerName="main" Sep 19 12:49:02.626667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.625148 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="582de267-73ae-458f-8d27-34667310ffd8" containerName="main" Sep 19 12:49:02.626667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.625233 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="582de267-73ae-458f-8d27-34667310ffd8" containerName="main" Sep 19 12:49:02.626667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.625245 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b123fea1-b0af-48a8-9b7e-b99aac6efccb" containerName="maas-api" Sep 19 12:49:02.628319 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.628302 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.635378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.635283 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d6124\"/\"auth-test-model-5d6124-kserve-self-signed-certs\"" Sep 19 12:49:02.635867 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.635845 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d6124\"/\"default-dockercfg-2cfx6\"" Sep 19 12:49:02.635963 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.635849 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d6124\"/\"kube-root-ca.crt\"" Sep 19 12:49:02.635963 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.635849 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d6124\"/\"openshift-service-ca.crt\"" Sep 19 12:49:02.645176 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.645153 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq"] Sep 19 12:49:02.765632 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.765596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7w8d\" (UniqueName: \"kubernetes.io/projected/67019302-2c92-448c-b05f-60f42c531a95-kube-api-access-k7w8d\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.765844 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.765642 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67019302-2c92-448c-b05f-60f42c531a95-tls-certs\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.765844 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.765765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-tmp-dir\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.765976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.765871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-home\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.765976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.765902 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-dshm\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.765976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.765960 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-model-cache\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867188 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67019302-2c92-448c-b05f-60f42c531a95-tls-certs\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-tmp-dir\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-home\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-dshm\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867347 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-model-cache\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867556 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7w8d\" (UniqueName: \"kubernetes.io/projected/67019302-2c92-448c-b05f-60f42c531a95-kube-api-access-k7w8d\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867751 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867718 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-home\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867900 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867754 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-tmp-dir\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.867900 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.867872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-model-cache\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.870414 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.870260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-dshm\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.870953 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.870932 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67019302-2c92-448c-b05f-60f42c531a95-tls-certs\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.883708 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.883637 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7w8d\" (UniqueName: \"kubernetes.io/projected/67019302-2c92-448c-b05f-60f42c531a95-kube-api-access-k7w8d\") pod \"auth-test-model-5d6124-kserve-86f5799f47-nnlzq\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:02.938540 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:02.938507 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:03.215817 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:03.215792 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq"] Sep 19 12:49:03.217165 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:49:03.217139 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67019302_2c92_448c_b05f_60f42c531a95.slice/crio-74ec750d23218f25a38c1f2d7cc91e9dc8cb9c4a1475b18d5fa3efc730076640 WatchSource:0}: Error finding container 74ec750d23218f25a38c1f2d7cc91e9dc8cb9c4a1475b18d5fa3efc730076640: Status 404 returned error can't find the container with id 74ec750d23218f25a38c1f2d7cc91e9dc8cb9c4a1475b18d5fa3efc730076640 Sep 19 12:49:03.807008 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:03.806971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" event={"ID":"67019302-2c92-448c-b05f-60f42c531a95","Type":"ContainerStarted","Data":"58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24"} Sep 19 12:49:03.807385 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:03.807015 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" event={"ID":"67019302-2c92-448c-b05f-60f42c531a95","Type":"ContainerStarted","Data":"74ec750d23218f25a38c1f2d7cc91e9dc8cb9c4a1475b18d5fa3efc730076640"} Sep 19 12:49:03.826185 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:03.826131 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" podStartSLOduration=1.826117174 podStartE2EDuration="1.826117174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:49:03.823054808 +0000 UTC m=+1439.429905388" watchObservedRunningTime="2026-09-19 12:49:03.826117174 +0000 UTC m=+1439.432967752" Sep 19 12:49:12.939114 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:12.939076 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:12.939519 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:12.939141 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:12.951786 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:12.951764 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:49:13.885344 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:49:13.885307 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:50:06.651679 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:06.651651 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:50:06.654998 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:06.654976 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:50:52.261073 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.261040 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz"] Sep 19 12:50:52.265423 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.265399 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.268836 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.267804 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-49e81a-serving-cert\"" Sep 19 12:50:52.276165 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.276142 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz"] Sep 19 12:50:52.328103 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.328080 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-maas-api-tls\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.328228 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.328139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jtw\" (UniqueName: \"kubernetes.io/projected/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-kube-api-access-r6jtw\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.328228 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.328195 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-openshift-service-ca\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.328309 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.328236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-metrics-certs\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.429297 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.429268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-openshift-service-ca\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.429469 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.429315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-metrics-certs\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.429469 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.429389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-maas-api-tls\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.429469 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.429444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jtw\" (UniqueName: \"kubernetes.io/projected/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-kube-api-access-r6jtw\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.429998 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.429973 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-openshift-service-ca\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.431924 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.431903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-metrics-certs\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.432009 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.431982 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-maas-api-tls\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.437052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.437032 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jtw\" (UniqueName: \"kubernetes.io/projected/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-kube-api-access-r6jtw\") pod \"maas-api-e2e-api-a-49e81a-565b66b4df-7dptz\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.583948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.583919 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:52.868216 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.868174 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz"] Sep 19 12:50:52.870061 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:50:52.870013 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5285d8_0b4b_4d30_ad7d_a0ca7e3c1731.slice/crio-40847cc5a6e1faf0b97283b3508ef6b991ad800bc911d562b35125ac79f9f8db WatchSource:0}: Error finding container 40847cc5a6e1faf0b97283b3508ef6b991ad800bc911d562b35125ac79f9f8db: Status 404 returned error can't find the container with id 40847cc5a6e1faf0b97283b3508ef6b991ad800bc911d562b35125ac79f9f8db Sep 19 12:50:52.871810 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:52.871787 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 12:50:53.548911 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:53.548879 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" event={"ID":"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731","Type":"ContainerStarted","Data":"68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5"} Sep 19 12:50:53.549269 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:53.548929 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" event={"ID":"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731","Type":"ContainerStarted","Data":"40847cc5a6e1faf0b97283b3508ef6b991ad800bc911d562b35125ac79f9f8db"} Sep 19 12:50:53.549269 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:53.548949 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:50:53.567154 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:53.567115 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" podStartSLOduration=0.988640719 podStartE2EDuration="1.56710026s" totalImagesPullingTime="578.459541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:50:53.566246119 +0000 UTC m=+1549.173096716" watchObservedRunningTime="2026-09-19 12:50:53.56710026 +0000 UTC m=+1549.173950839" Sep 19 12:50:59.561056 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:50:59.561028 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:51:23.147601 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.147554 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf"] Sep 19 12:51:23.152726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.152662 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.155977 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.155945 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-513a51-serving-cert\"" Sep 19 12:51:23.164823 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.164798 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf"] Sep 19 12:51:23.249177 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.249138 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.249370 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.249220 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-metrics-certs\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.249370 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.249343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l75v\" (UniqueName: \"kubernetes.io/projected/77f8ad78-b477-4ad4-b012-3f47889a9c3f-kube-api-access-5l75v\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.249527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.249416 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f8ad78-b477-4ad4-b012-3f47889a9c3f-openshift-service-ca\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.350585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.350543 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l75v\" (UniqueName: \"kubernetes.io/projected/77f8ad78-b477-4ad4-b012-3f47889a9c3f-kube-api-access-5l75v\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.350585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.350596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f8ad78-b477-4ad4-b012-3f47889a9c3f-openshift-service-ca\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.350862 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.350653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.350862 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.350719 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-metrics-certs\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.350984 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:51:23.350863 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-serving-cert: secret "maas-api-e2e-api-b-513a51-serving-cert" not found Sep 19 12:51:23.350984 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:51:23.350937 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls podName:77f8ad78-b477-4ad4-b012-3f47889a9c3f nodeName:}" failed. No retries permitted until 2026-09-19 12:51:23.85091795 +0000 UTC m=+1579.457768522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls") pod "maas-api-e2e-api-b-513a51-59c75b7956-nswlf" (UID: "77f8ad78-b477-4ad4-b012-3f47889a9c3f") : secret "maas-api-e2e-api-b-513a51-serving-cert" not found Sep 19 12:51:23.351363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.351341 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f8ad78-b477-4ad4-b012-3f47889a9c3f-openshift-service-ca\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.353210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.353194 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-metrics-certs\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.361811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.361788 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l75v\" (UniqueName: \"kubernetes.io/projected/77f8ad78-b477-4ad4-b012-3f47889a9c3f-kube-api-access-5l75v\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.856553 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.856522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:23.859062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:23.859037 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls\") pod \"maas-api-e2e-api-b-513a51-59c75b7956-nswlf\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:24.071357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:24.071316 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:24.356351 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:24.356306 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf"] Sep 19 12:51:24.357529 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:51:24.357505 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f8ad78_b477_4ad4_b012_3f47889a9c3f.slice/crio-94cb8f5ab6668f846fe8f638d9967740b01fb840d424ae59bd2d1da282a092de WatchSource:0}: Error finding container 94cb8f5ab6668f846fe8f638d9967740b01fb840d424ae59bd2d1da282a092de: Status 404 returned error can't find the container with id 94cb8f5ab6668f846fe8f638d9967740b01fb840d424ae59bd2d1da282a092de Sep 19 12:51:24.775533 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:24.775500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" event={"ID":"77f8ad78-b477-4ad4-b012-3f47889a9c3f","Type":"ContainerStarted","Data":"daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3"} Sep 19 12:51:24.775721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:24.775557 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:24.775721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:24.775574 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" event={"ID":"77f8ad78-b477-4ad4-b012-3f47889a9c3f","Type":"ContainerStarted","Data":"94cb8f5ab6668f846fe8f638d9967740b01fb840d424ae59bd2d1da282a092de"} Sep 19 12:51:24.792968 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:24.792915 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" podStartSLOduration=1.503472363 podStartE2EDuration="1.792898593s" totalImagesPullingTime="289.42623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:51:24.791817433 +0000 UTC m=+1580.398668007" watchObservedRunningTime="2026-09-19 12:51:24.792898593 +0000 UTC m=+1580.399749172" Sep 19 12:51:27.799064 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:27.799032 2585 generic.go:356] "Generic (PLEG): container finished" podID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerID="ca0f381632d0760bf1e40a0de38b0f421fe4b2252316a1e93e722df1967ec161" exitCode=137 Sep 19 12:51:27.799468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:27.799102 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" event={"ID":"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4","Type":"ContainerDied","Data":"ca0f381632d0760bf1e40a0de38b0f421fe4b2252316a1e93e722df1967ec161"} Sep 19 12:51:27.799468 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:27.799433 2585 scope.go:117] "RemoveContainer" containerID="ca0f381632d0760bf1e40a0de38b0f421fe4b2252316a1e93e722df1967ec161" Sep 19 12:51:28.807413 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:28.807378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" event={"ID":"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4","Type":"ContainerStarted","Data":"c41845aafc9465ee5b916f803c287ac03ec734fdc4df6461af707d552f915823"} Sep 19 12:51:28.807846 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:28.807601 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:51:29.134923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.134847 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k"] Sep 19 12:51:29.135675 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.135145 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" podUID="5720c5e4-7bcf-4881-9a62-28e38d60e042" containerName="maas-api" containerID="cri-o://2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9" gracePeriod=30 Sep 19 12:51:29.413505 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.413476 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:51:29.523557 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.523524 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-metrics-certs\") pod \"5720c5e4-7bcf-4881-9a62-28e38d60e042\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " Sep 19 12:51:29.523557 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.523565 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhr46\" (UniqueName: \"kubernetes.io/projected/5720c5e4-7bcf-4881-9a62-28e38d60e042-kube-api-access-fhr46\") pod \"5720c5e4-7bcf-4881-9a62-28e38d60e042\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " Sep 19 12:51:29.523865 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.523628 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls\") pod \"5720c5e4-7bcf-4881-9a62-28e38d60e042\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " Sep 19 12:51:29.523865 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.523823 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5720c5e4-7bcf-4881-9a62-28e38d60e042-openshift-service-ca\") pod \"5720c5e4-7bcf-4881-9a62-28e38d60e042\" (UID: \"5720c5e4-7bcf-4881-9a62-28e38d60e042\") " Sep 19 12:51:29.524567 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.524537 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5720c5e4-7bcf-4881-9a62-28e38d60e042-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5720c5e4-7bcf-4881-9a62-28e38d60e042" (UID: "5720c5e4-7bcf-4881-9a62-28e38d60e042"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:51:29.526083 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.526061 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5720c5e4-7bcf-4881-9a62-28e38d60e042-kube-api-access-fhr46" (OuterVolumeSpecName: "kube-api-access-fhr46") pod "5720c5e4-7bcf-4881-9a62-28e38d60e042" (UID: "5720c5e4-7bcf-4881-9a62-28e38d60e042"). InnerVolumeSpecName "kube-api-access-fhr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:51:29.526356 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.526339 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5720c5e4-7bcf-4881-9a62-28e38d60e042" (UID: "5720c5e4-7bcf-4881-9a62-28e38d60e042"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:51:29.526535 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.526522 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5720c5e4-7bcf-4881-9a62-28e38d60e042" (UID: "5720c5e4-7bcf-4881-9a62-28e38d60e042"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:51:29.625568 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.625528 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:29.625568 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.625564 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhr46\" (UniqueName: \"kubernetes.io/projected/5720c5e4-7bcf-4881-9a62-28e38d60e042-kube-api-access-fhr46\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:29.625568 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.625576 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5720c5e4-7bcf-4881-9a62-28e38d60e042-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:29.625850 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.625587 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5720c5e4-7bcf-4881-9a62-28e38d60e042-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:29.814819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.814787 2585 generic.go:356] "Generic (PLEG): container finished" podID="5720c5e4-7bcf-4881-9a62-28e38d60e042" containerID="2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9" exitCode=0 Sep 19 12:51:29.815251 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.814829 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" Sep 19 12:51:29.815251 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.814869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" event={"ID":"5720c5e4-7bcf-4881-9a62-28e38d60e042","Type":"ContainerDied","Data":"2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9"} Sep 19 12:51:29.815251 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.814917 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k" event={"ID":"5720c5e4-7bcf-4881-9a62-28e38d60e042","Type":"ContainerDied","Data":"caf57563481f15f7d2feb9dd8fb62de7d603e395c9d3d8f59d8091833596f882"} Sep 19 12:51:29.815251 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.814945 2585 scope.go:117] "RemoveContainer" containerID="2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9" Sep 19 12:51:29.827236 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.827221 2585 scope.go:117] "RemoveContainer" containerID="2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9" Sep 19 12:51:29.827508 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:51:29.827484 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9\": container with ID starting with 2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9 not found: ID does not exist" containerID="2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9" Sep 19 12:51:29.827557 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.827520 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9"} err="failed to get container status \"2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9\": rpc error: code = NotFound desc = could not find container \"2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9\": container with ID starting with 2a3c93bbc7028a0263808ce8ab2ee30b0f35ba5fd905268f9823003849da99f9 not found: ID does not exist" Sep 19 12:51:29.841250 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.841227 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k"] Sep 19 12:51:29.862043 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:29.862024 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a2e924-7bdb547d6d-hxm7k"] Sep 19 12:51:30.579928 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:30.579899 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5720c5e4-7bcf-4881-9a62-28e38d60e042" path="/var/lib/kubelet/pods/5720c5e4-7bcf-4881-9a62-28e38d60e042/volumes" Sep 19 12:51:30.788832 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:30.788805 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:51:36.526529 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:36.526490 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw"] Sep 19 12:51:36.527089 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:36.527058 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" containerID="cri-o://25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d" gracePeriod=30 Sep 19 12:51:39.817471 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:39.817435 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 12:51:41.797883 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.797825 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:51:41.847421 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847394 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57b4k\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-kube-api-access-57b4k\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847442 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-socket\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847481 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istiod-ca-cert\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847519 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-podinfo\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847593 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847558 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-data\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847854 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847610 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-certs\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847854 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847643 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-credential-socket\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847854 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847745 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-envoy\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.847854 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847779 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-token\") pod \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\" (UID: \"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687\") " Sep 19 12:51:41.848037 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847858 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:51:41.848037 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847877 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:51:41.848037 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847981 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:51:41.848037 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.847988 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:51:41.848266 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.848078 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:51:41.848381 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.848358 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-data\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.848445 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.848387 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.848517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.848462 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-credential-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.848591 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.848569 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-workload-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.848779 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.848596 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istiod-ca-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.850173 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.850143 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-kube-api-access-57b4k" (OuterVolumeSpecName: "kube-api-access-57b4k") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "kube-api-access-57b4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:51:41.850383 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.850360 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 12:51:41.850383 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.850374 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:51:41.850557 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.850373 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" (UID: "dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:51:41.898536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.898507 2585 generic.go:356] "Generic (PLEG): container finished" podID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerID="25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d" exitCode=0 Sep 19 12:51:41.898658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.898558 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" Sep 19 12:51:41.898658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.898568 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" event={"ID":"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687","Type":"ContainerDied","Data":"25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d"} Sep 19 12:51:41.898658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.898626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw" event={"ID":"dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687","Type":"ContainerDied","Data":"092b9a5b0b443b306b1cf4f7ad5cecd59016d5a0c530fc26b39b2ab36fe4de44"} Sep 19 12:51:41.898658 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.898650 2585 scope.go:117] "RemoveContainer" containerID="25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d" Sep 19 12:51:41.909399 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.909379 2585 scope.go:117] "RemoveContainer" containerID="b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882" Sep 19 12:51:41.919187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.919072 2585 scope.go:117] "RemoveContainer" containerID="25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d" Sep 19 12:51:41.919386 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:51:41.919367 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d\": container with ID starting with 25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d not found: ID does not exist" containerID="25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d" Sep 19 12:51:41.919457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.919399 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d"} err="failed to get container status \"25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d\": rpc error: code = NotFound desc = could not find container \"25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d\": container with ID starting with 25533805af5df1876eff3b7fc878ac93432c6d83af65be99ba418beef324df6d not found: ID does not exist" Sep 19 12:51:41.919457 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.919423 2585 scope.go:117] "RemoveContainer" containerID="b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882" Sep 19 12:51:41.919721 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:51:41.919701 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882\": container with ID starting with b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882 not found: ID does not exist" containerID="b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882" Sep 19 12:51:41.919773 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.919728 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882"} err="failed to get container status \"b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882\": rpc error: code = NotFound desc = could not find container \"b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882\": container with ID starting with b989e4775e7abd2e7ed8ecd5dad1df04c8d31419e383c7ec8142aedfd16c3882 not found: ID does not exist" Sep 19 12:51:41.921106 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.921088 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw"] Sep 19 12:51:41.927252 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.927233 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a2e924-openshift-default-5bb89d9d6b-6t2jw"] Sep 19 12:51:41.949318 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.949291 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-envoy\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.949318 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.949317 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-token\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.949447 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.949330 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57b4k\" (UniqueName: \"kubernetes.io/projected/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-kube-api-access-57b4k\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:41.949447 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:41.949343 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687-istio-podinfo\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:42.581000 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:42.580958 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" path="/var/lib/kubelet/pods/dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687/volumes" Sep 19 12:51:59.466672 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.466574 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz"] Sep 19 12:51:59.467244 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.466901 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" podUID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" containerName="maas-api" containerID="cri-o://68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5" gracePeriod=30 Sep 19 12:51:59.556480 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.556195 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" podUID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.88:8443/health\": dial tcp 10.132.0.88:8443: connect: connection refused" Sep 19 12:51:59.762244 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.762220 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:51:59.834146 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.833267 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-metrics-certs\") pod \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " Sep 19 12:51:59.834146 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.833345 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6jtw\" (UniqueName: \"kubernetes.io/projected/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-kube-api-access-r6jtw\") pod \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " Sep 19 12:51:59.834146 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.833448 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-maas-api-tls\") pod \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " Sep 19 12:51:59.834146 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.833519 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-openshift-service-ca\") pod \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\" (UID: \"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731\") " Sep 19 12:51:59.835004 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.834670 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" (UID: "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:51:59.837406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.837377 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" (UID: "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:51:59.837595 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.837575 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-kube-api-access-r6jtw" (OuterVolumeSpecName: "kube-api-access-r6jtw") pod "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" (UID: "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731"). InnerVolumeSpecName "kube-api-access-r6jtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:51:59.838406 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.838378 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" (UID: "7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:51:59.935025 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.934986 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:59.935025 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.935016 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jtw\" (UniqueName: \"kubernetes.io/projected/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-kube-api-access-r6jtw\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:59.935025 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.935026 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:51:59.935536 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:51:59.935037 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:00.021011 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.020912 2585 generic.go:356] "Generic (PLEG): container finished" podID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" containerID="68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5" exitCode=0 Sep 19 12:52:00.021011 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.020980 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" Sep 19 12:52:00.021233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.021013 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" event={"ID":"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731","Type":"ContainerDied","Data":"68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5"} Sep 19 12:52:00.021233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.021076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz" event={"ID":"7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731","Type":"ContainerDied","Data":"40847cc5a6e1faf0b97283b3508ef6b991ad800bc911d562b35125ac79f9f8db"} Sep 19 12:52:00.021233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.021105 2585 scope.go:117] "RemoveContainer" containerID="68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5" Sep 19 12:52:00.031518 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.031500 2585 scope.go:117] "RemoveContainer" containerID="68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5" Sep 19 12:52:00.031830 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:52:00.031810 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5\": container with ID starting with 68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5 not found: ID does not exist" containerID="68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5" Sep 19 12:52:00.031912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.031838 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5"} err="failed to get container status \"68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5\": rpc error: code = NotFound desc = could not find container \"68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5\": container with ID starting with 68a0095572bef47e35a598969a4c3f584124b41c96010e2d9871cb918d53b9d5 not found: ID does not exist" Sep 19 12:52:00.043652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.043628 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz"] Sep 19 12:52:00.050075 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.050052 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49e81a-565b66b4df-7dptz"] Sep 19 12:52:00.583503 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:00.583456 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" path="/var/lib/kubelet/pods/7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731/volumes" Sep 19 12:52:23.085222 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.085187 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf"] Sep 19 12:52:23.085723 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.085513 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" podUID="77f8ad78-b477-4ad4-b012-3f47889a9c3f" containerName="maas-api" containerID="cri-o://daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3" gracePeriod=30 Sep 19 12:52:23.344518 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.344459 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:52:23.359625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.359601 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l75v\" (UniqueName: \"kubernetes.io/projected/77f8ad78-b477-4ad4-b012-3f47889a9c3f-kube-api-access-5l75v\") pod \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " Sep 19 12:52:23.359790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.359711 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f8ad78-b477-4ad4-b012-3f47889a9c3f-openshift-service-ca\") pod \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " Sep 19 12:52:23.359790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.359748 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-metrics-certs\") pod \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " Sep 19 12:52:23.359790 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.359783 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls\") pod \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\" (UID: \"77f8ad78-b477-4ad4-b012-3f47889a9c3f\") " Sep 19 12:52:23.360134 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.360101 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f8ad78-b477-4ad4-b012-3f47889a9c3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77f8ad78-b477-4ad4-b012-3f47889a9c3f" (UID: "77f8ad78-b477-4ad4-b012-3f47889a9c3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:52:23.360274 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.360225 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f8ad78-b477-4ad4-b012-3f47889a9c3f-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:23.362487 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.362443 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77f8ad78-b477-4ad4-b012-3f47889a9c3f" (UID: "77f8ad78-b477-4ad4-b012-3f47889a9c3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:52:23.362782 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.362746 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f8ad78-b477-4ad4-b012-3f47889a9c3f-kube-api-access-5l75v" (OuterVolumeSpecName: "kube-api-access-5l75v") pod "77f8ad78-b477-4ad4-b012-3f47889a9c3f" (UID: "77f8ad78-b477-4ad4-b012-3f47889a9c3f"). InnerVolumeSpecName "kube-api-access-5l75v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:52:23.363358 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.363333 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "77f8ad78-b477-4ad4-b012-3f47889a9c3f" (UID: "77f8ad78-b477-4ad4-b012-3f47889a9c3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:52:23.461910 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.461879 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l75v\" (UniqueName: \"kubernetes.io/projected/77f8ad78-b477-4ad4-b012-3f47889a9c3f-kube-api-access-5l75v\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:23.462040 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.461916 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:23.462040 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:23.461930 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f8ad78-b477-4ad4-b012-3f47889a9c3f-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:24.191285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.191251 2585 generic.go:356] "Generic (PLEG): container finished" podID="77f8ad78-b477-4ad4-b012-3f47889a9c3f" containerID="daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3" exitCode=0 Sep 19 12:52:24.191742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.191297 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" Sep 19 12:52:24.191742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.191339 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" event={"ID":"77f8ad78-b477-4ad4-b012-3f47889a9c3f","Type":"ContainerDied","Data":"daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3"} Sep 19 12:52:24.191742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.191386 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf" event={"ID":"77f8ad78-b477-4ad4-b012-3f47889a9c3f","Type":"ContainerDied","Data":"94cb8f5ab6668f846fe8f638d9967740b01fb840d424ae59bd2d1da282a092de"} Sep 19 12:52:24.191742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.191408 2585 scope.go:117] "RemoveContainer" containerID="daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3" Sep 19 12:52:24.202625 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.202610 2585 scope.go:117] "RemoveContainer" containerID="daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3" Sep 19 12:52:24.202956 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:52:24.202940 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3\": container with ID starting with daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3 not found: ID does not exist" containerID="daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3" Sep 19 12:52:24.203013 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.202964 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3"} err="failed to get container status \"daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3\": rpc error: code = NotFound desc = could not find container \"daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3\": container with ID starting with daa1f28639fdf7233cfcd8450443532a5f57d00e0dd18968491a9e9c414367a3 not found: ID does not exist" Sep 19 12:52:24.212936 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.212911 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf"] Sep 19 12:52:24.216675 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.216656 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-513a51-59c75b7956-nswlf"] Sep 19 12:52:24.580866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:24.580835 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f8ad78-b477-4ad4-b012-3f47889a9c3f" path="/var/lib/kubelet/pods/77f8ad78-b477-4ad4-b012-3f47889a9c3f/volumes" Sep 19 12:52:40.823588 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:40.823552 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb"] Sep 19 12:52:40.824166 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:40.823863 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" podUID="f67a11cc-ba15-45d8-8fcb-740a2b382608" containerName="maas-api" containerID="cri-o://49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890" gracePeriod=30 Sep 19 12:52:41.083483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.083426 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:52:41.235610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.235576 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkz2c\" (UniqueName: \"kubernetes.io/projected/f67a11cc-ba15-45d8-8fcb-740a2b382608-kube-api-access-wkz2c\") pod \"f67a11cc-ba15-45d8-8fcb-740a2b382608\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " Sep 19 12:52:41.235827 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.235701 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67a11cc-ba15-45d8-8fcb-740a2b382608-openshift-service-ca\") pod \"f67a11cc-ba15-45d8-8fcb-740a2b382608\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " Sep 19 12:52:41.235827 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.235771 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-metrics-certs\") pod \"f67a11cc-ba15-45d8-8fcb-740a2b382608\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " Sep 19 12:52:41.235963 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.235879 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls\") pod \"f67a11cc-ba15-45d8-8fcb-740a2b382608\" (UID: \"f67a11cc-ba15-45d8-8fcb-740a2b382608\") " Sep 19 12:52:41.236354 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.236324 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67a11cc-ba15-45d8-8fcb-740a2b382608-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f67a11cc-ba15-45d8-8fcb-740a2b382608" (UID: "f67a11cc-ba15-45d8-8fcb-740a2b382608"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:52:41.238241 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.238212 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f67a11cc-ba15-45d8-8fcb-740a2b382608" (UID: "f67a11cc-ba15-45d8-8fcb-740a2b382608"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:52:41.238719 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.238672 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67a11cc-ba15-45d8-8fcb-740a2b382608-kube-api-access-wkz2c" (OuterVolumeSpecName: "kube-api-access-wkz2c") pod "f67a11cc-ba15-45d8-8fcb-740a2b382608" (UID: "f67a11cc-ba15-45d8-8fcb-740a2b382608"). InnerVolumeSpecName "kube-api-access-wkz2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:52:41.239100 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.239074 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f67a11cc-ba15-45d8-8fcb-740a2b382608" (UID: "f67a11cc-ba15-45d8-8fcb-740a2b382608"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:52:41.307110 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.307073 2585 generic.go:356] "Generic (PLEG): container finished" podID="f67a11cc-ba15-45d8-8fcb-740a2b382608" containerID="49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890" exitCode=0 Sep 19 12:52:41.307268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.307134 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" Sep 19 12:52:41.307268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.307160 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" event={"ID":"f67a11cc-ba15-45d8-8fcb-740a2b382608","Type":"ContainerDied","Data":"49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890"} Sep 19 12:52:41.307268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.307219 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb" event={"ID":"f67a11cc-ba15-45d8-8fcb-740a2b382608","Type":"ContainerDied","Data":"45ef8414a636d218d76186bf9eae9961493bec33ed9d4d658315c1f6e999f6a1"} Sep 19 12:52:41.307268 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.307251 2585 scope.go:117] "RemoveContainer" containerID="49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890" Sep 19 12:52:41.318150 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.317990 2585 scope.go:117] "RemoveContainer" containerID="49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890" Sep 19 12:52:41.318344 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:52:41.318319 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890\": container with ID starting with 49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890 not found: ID does not exist" containerID="49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890" Sep 19 12:52:41.318445 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.318356 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890"} err="failed to get container status \"49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890\": rpc error: code = NotFound desc = could not find container \"49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890\": container with ID starting with 49f8c25f9d0c4f7633f254a781f388c4beb5e193c1e98976667dfe504aad9890 not found: ID does not exist" Sep 19 12:52:41.333942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.333889 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb"] Sep 19 12:52:41.337374 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.337354 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:41.337374 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.337374 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67a11cc-ba15-45d8-8fcb-740a2b382608-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:41.337513 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.337383 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkz2c\" (UniqueName: \"kubernetes.io/projected/f67a11cc-ba15-45d8-8fcb-740a2b382608-kube-api-access-wkz2c\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:41.337513 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.337392 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67a11cc-ba15-45d8-8fcb-740a2b382608-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:41.339008 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:41.338990 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a487c-c998cb8fc-kbhbb"] Sep 19 12:52:42.581678 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:42.581647 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67a11cc-ba15-45d8-8fcb-740a2b382608" path="/var/lib/kubelet/pods/f67a11cc-ba15-45d8-8fcb-740a2b382608/volumes" Sep 19 12:52:58.919528 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:58.919491 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56"] Sep 19 12:52:58.920016 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:58.919785 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" podUID="70686a87-7638-4a87-bb58-fbfdac6a6b6a" containerName="maas-api" containerID="cri-o://d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d" gracePeriod=30 Sep 19 12:52:59.180769 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.180679 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:52:59.209463 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.209434 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls\") pod \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " Sep 19 12:52:59.209725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.209471 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70686a87-7638-4a87-bb58-fbfdac6a6b6a-openshift-service-ca\") pod \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " Sep 19 12:52:59.209725 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.209606 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-metrics-certs\") pod \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " Sep 19 12:52:59.209890 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.209757 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngnkh\" (UniqueName: \"kubernetes.io/projected/70686a87-7638-4a87-bb58-fbfdac6a6b6a-kube-api-access-ngnkh\") pod \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\" (UID: \"70686a87-7638-4a87-bb58-fbfdac6a6b6a\") " Sep 19 12:52:59.209890 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.209824 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70686a87-7638-4a87-bb58-fbfdac6a6b6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70686a87-7638-4a87-bb58-fbfdac6a6b6a" (UID: "70686a87-7638-4a87-bb58-fbfdac6a6b6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:52:59.210150 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.210128 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70686a87-7638-4a87-bb58-fbfdac6a6b6a-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:59.211888 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.211865 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70686a87-7638-4a87-bb58-fbfdac6a6b6a" (UID: "70686a87-7638-4a87-bb58-fbfdac6a6b6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:52:59.212068 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.212044 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70686a87-7638-4a87-bb58-fbfdac6a6b6a" (UID: "70686a87-7638-4a87-bb58-fbfdac6a6b6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:52:59.212378 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.212359 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70686a87-7638-4a87-bb58-fbfdac6a6b6a-kube-api-access-ngnkh" (OuterVolumeSpecName: "kube-api-access-ngnkh") pod "70686a87-7638-4a87-bb58-fbfdac6a6b6a" (UID: "70686a87-7638-4a87-bb58-fbfdac6a6b6a"). InnerVolumeSpecName "kube-api-access-ngnkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:52:59.311462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.311431 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnkh\" (UniqueName: \"kubernetes.io/projected/70686a87-7638-4a87-bb58-fbfdac6a6b6a-kube-api-access-ngnkh\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:59.311462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.311460 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:59.311462 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.311470 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70686a87-7638-4a87-bb58-fbfdac6a6b6a-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:52:59.426875 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.426836 2585 generic.go:356] "Generic (PLEG): container finished" podID="70686a87-7638-4a87-bb58-fbfdac6a6b6a" containerID="d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d" exitCode=0 Sep 19 12:52:59.427034 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.426886 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" Sep 19 12:52:59.427034 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.426924 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" event={"ID":"70686a87-7638-4a87-bb58-fbfdac6a6b6a","Type":"ContainerDied","Data":"d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d"} Sep 19 12:52:59.427034 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.426980 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56" event={"ID":"70686a87-7638-4a87-bb58-fbfdac6a6b6a","Type":"ContainerDied","Data":"bdb83ee34b77561a3687822d5f07619888aafde40baa41d946290ad9a81947cf"} Sep 19 12:52:59.427034 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.427002 2585 scope.go:117] "RemoveContainer" containerID="d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d" Sep 19 12:52:59.437524 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.437508 2585 scope.go:117] "RemoveContainer" containerID="d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d" Sep 19 12:52:59.437821 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:52:59.437800 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d\": container with ID starting with d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d not found: ID does not exist" containerID="d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d" Sep 19 12:52:59.437895 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.437826 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d"} err="failed to get container status \"d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d\": rpc error: code = NotFound desc = could not find container \"d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d\": container with ID starting with d3aa05b5d4e5045d39407464f315bdb712b3be9c9f2c740c493033f3f560a40d not found: ID does not exist" Sep 19 12:52:59.451964 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.451939 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56"] Sep 19 12:52:59.462740 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:52:59.462711 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5068d9-67d6c75987-pkp56"] Sep 19 12:53:00.580259 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:00.580224 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70686a87-7638-4a87-bb58-fbfdac6a6b6a" path="/var/lib/kubelet/pods/70686a87-7638-4a87-bb58-fbfdac6a6b6a/volumes" Sep 19 12:53:09.347873 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.347841 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-04161d"] Sep 19 12:53:09.348325 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348311 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348326 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348333 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" containerName="maas-api" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348339 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" containerName="maas-api" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348349 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348355 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348366 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5720c5e4-7bcf-4881-9a62-28e38d60e042" containerName="maas-api" Sep 19 12:53:09.348369 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348370 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5720c5e4-7bcf-4881-9a62-28e38d60e042" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348384 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70686a87-7638-4a87-bb58-fbfdac6a6b6a" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348389 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="70686a87-7638-4a87-bb58-fbfdac6a6b6a" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348399 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f8ad78-b477-4ad4-b012-3f47889a9c3f" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348404 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f8ad78-b477-4ad4-b012-3f47889a9c3f" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348416 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67a11cc-ba15-45d8-8fcb-740a2b382608" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348421 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67a11cc-ba15-45d8-8fcb-740a2b382608" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348488 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5720c5e4-7bcf-4881-9a62-28e38d60e042" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348499 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5285d8-0b4b-4d30-ad7d-a0ca7e3c1731" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348506 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348512 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f8ad78-b477-4ad4-b012-3f47889a9c3f" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348519 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67a11cc-ba15-45d8-8fcb-740a2b382608" containerName="maas-api" Sep 19 12:53:09.348582 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.348526 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="70686a87-7638-4a87-bb58-fbfdac6a6b6a" containerName="maas-api" Sep 19 12:53:09.353161 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.353141 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:09.356278 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.356258 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pjdxd\"" Sep 19 12:53:09.369465 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.369441 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-04161d"] Sep 19 12:53:09.403787 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.403761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5k7\" (UniqueName: \"kubernetes.io/projected/64dbb6f1-1a19-4b10-8579-14a7a20ad562-kube-api-access-qk5k7\") pod \"test-curl-7567-04161d\" (UID: \"64dbb6f1-1a19-4b10-8579-14a7a20ad562\") " pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:09.505212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.505157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5k7\" (UniqueName: \"kubernetes.io/projected/64dbb6f1-1a19-4b10-8579-14a7a20ad562-kube-api-access-qk5k7\") pod \"test-curl-7567-04161d\" (UID: \"64dbb6f1-1a19-4b10-8579-14a7a20ad562\") " pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:09.514195 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.514166 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5k7\" (UniqueName: \"kubernetes.io/projected/64dbb6f1-1a19-4b10-8579-14a7a20ad562-kube-api-access-qk5k7\") pod \"test-curl-7567-04161d\" (UID: \"64dbb6f1-1a19-4b10-8579-14a7a20ad562\") " pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:09.661866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.661813 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:09.936420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:09.936394 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-04161d"] Sep 19 12:53:09.937779 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:53:09.937752 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64dbb6f1_1a19_4b10_8579_14a7a20ad562.slice/crio-633fc057bd98e1c7744c5496e76a05d85a76c658f8c59c7d1c4f4a299ea657c7 WatchSource:0}: Error finding container 633fc057bd98e1c7744c5496e76a05d85a76c658f8c59c7d1c4f4a299ea657c7: Status 404 returned error can't find the container with id 633fc057bd98e1c7744c5496e76a05d85a76c658f8c59c7d1c4f4a299ea657c7 Sep 19 12:53:10.500517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:10.500469 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-04161d" event={"ID":"64dbb6f1-1a19-4b10-8579-14a7a20ad562","Type":"ContainerStarted","Data":"68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce"} Sep 19 12:53:10.500517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:10.500519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-04161d" event={"ID":"64dbb6f1-1a19-4b10-8579-14a7a20ad562","Type":"ContainerStarted","Data":"633fc057bd98e1c7744c5496e76a05d85a76c658f8c59c7d1c4f4a299ea657c7"} Sep 19 12:53:10.519551 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:10.519498 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7567-04161d" podStartSLOduration=1.519486755 podStartE2EDuration="1.519486755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:53:10.518448007 +0000 UTC m=+1686.125298589" watchObservedRunningTime="2026-09-19 12:53:10.519486755 +0000 UTC m=+1686.126337333" Sep 19 12:53:11.122845 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:11.122740 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-04161d"] Sep 19 12:53:11.126806 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:11.126780 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-04161d"] Sep 19 12:53:11.506561 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:11.506534 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7567-04161d" container="test-curl-7567-04161d" Sep 19 12:53:12.513222 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:12.513169 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7567-04161d" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" containerName="test-curl-7567-04161d" containerID="cri-o://68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce" gracePeriod=2 Sep 19 12:53:12.515257 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:12.515225 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-04161d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:13.924926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:13.924890 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq"] Sep 19 12:53:14.723542 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:14.723519 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:14.857452 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:14.857425 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk5k7\" (UniqueName: \"kubernetes.io/projected/64dbb6f1-1a19-4b10-8579-14a7a20ad562-kube-api-access-qk5k7\") pod \"64dbb6f1-1a19-4b10-8579-14a7a20ad562\" (UID: \"64dbb6f1-1a19-4b10-8579-14a7a20ad562\") " Sep 19 12:53:14.859561 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:14.859534 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dbb6f1-1a19-4b10-8579-14a7a20ad562-kube-api-access-qk5k7" (OuterVolumeSpecName: "kube-api-access-qk5k7") pod "64dbb6f1-1a19-4b10-8579-14a7a20ad562" (UID: "64dbb6f1-1a19-4b10-8579-14a7a20ad562"). InnerVolumeSpecName "kube-api-access-qk5k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:14.959309 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:14.959286 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5k7\" (UniqueName: \"kubernetes.io/projected/64dbb6f1-1a19-4b10-8579-14a7a20ad562-kube-api-access-qk5k7\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:15.534188 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:15.534154 2585 generic.go:356] "Generic (PLEG): container finished" podID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" containerID="68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce" exitCode=137 Sep 19 12:53:15.534390 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:15.534196 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:15.534390 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:15.534222 2585 scope.go:117] "RemoveContainer" containerID="68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce" Sep 19 12:53:15.543907 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:15.543887 2585 scope.go:117] "RemoveContainer" containerID="68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce" Sep 19 12:53:15.544172 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:15.544154 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce\": container with ID starting with 68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce not found: ID does not exist" containerID="68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce" Sep 19 12:53:15.544230 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:15.544180 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce"} err="failed to get container status \"68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce\": rpc error: code = NotFound desc = could not find container \"68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce\": container with ID starting with 68fcea35499bdcb432a98d271dfe8d98429d8aba8a679aaf9088a0926efe70ce not found: ID does not exist" Sep 19 12:53:15.544300 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:15.544285 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-04161d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:16.576907 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:16.576863 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-04161d\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" pod="opendatahub/test-curl-7567-04161d" Sep 19 12:53:16.580365 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:16.580336 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" path="/var/lib/kubelet/pods/64dbb6f1-1a19-4b10-8579-14a7a20ad562/volumes" Sep 19 12:53:28.945355 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:28.945283 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" podUID="67019302-2c92-448c-b05f-60f42c531a95" containerName="main" containerID="cri-o://58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24" gracePeriod=45 Sep 19 12:53:29.220652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.220613 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:53:29.384570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.384532 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx"] Sep 19 12:53:29.412667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412635 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-tmp-dir\") pod \"67019302-2c92-448c-b05f-60f42c531a95\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " Sep 19 12:53:29.412667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412677 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-model-cache\") pod \"67019302-2c92-448c-b05f-60f42c531a95\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " Sep 19 12:53:29.412948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412739 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67019302-2c92-448c-b05f-60f42c531a95-tls-certs\") pod \"67019302-2c92-448c-b05f-60f42c531a95\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " Sep 19 12:53:29.412948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412781 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-dshm\") pod \"67019302-2c92-448c-b05f-60f42c531a95\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " Sep 19 12:53:29.412948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412870 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-home\") pod \"67019302-2c92-448c-b05f-60f42c531a95\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " Sep 19 12:53:29.412948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412916 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7w8d\" (UniqueName: \"kubernetes.io/projected/67019302-2c92-448c-b05f-60f42c531a95-kube-api-access-k7w8d\") pod \"67019302-2c92-448c-b05f-60f42c531a95\" (UID: \"67019302-2c92-448c-b05f-60f42c531a95\") " Sep 19 12:53:29.412948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.412917 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67019302-2c92-448c-b05f-60f42c531a95" (UID: "67019302-2c92-448c-b05f-60f42c531a95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:29.413206 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.413027 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-model-cache" (OuterVolumeSpecName: "model-cache") pod "67019302-2c92-448c-b05f-60f42c531a95" (UID: "67019302-2c92-448c-b05f-60f42c531a95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:29.413260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.413196 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-home" (OuterVolumeSpecName: "home") pod "67019302-2c92-448c-b05f-60f42c531a95" (UID: "67019302-2c92-448c-b05f-60f42c531a95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:29.413376 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.413351 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-tmp-dir\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:29.413376 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.413375 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-model-cache\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:29.413785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.413383 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-home\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:29.415483 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.415455 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67019302-2c92-448c-b05f-60f42c531a95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67019302-2c92-448c-b05f-60f42c531a95" (UID: "67019302-2c92-448c-b05f-60f42c531a95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:53:29.415819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.415797 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-dshm" (OuterVolumeSpecName: "dshm") pod "67019302-2c92-448c-b05f-60f42c531a95" (UID: "67019302-2c92-448c-b05f-60f42c531a95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:29.415819 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.415804 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67019302-2c92-448c-b05f-60f42c531a95-kube-api-access-k7w8d" (OuterVolumeSpecName: "kube-api-access-k7w8d") pod "67019302-2c92-448c-b05f-60f42c531a95" (UID: "67019302-2c92-448c-b05f-60f42c531a95"). InnerVolumeSpecName "kube-api-access-k7w8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:29.514957 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.514862 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7w8d\" (UniqueName: \"kubernetes.io/projected/67019302-2c92-448c-b05f-60f42c531a95-kube-api-access-k7w8d\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:29.514957 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.514897 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67019302-2c92-448c-b05f-60f42c531a95-tls-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:29.514957 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.514908 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67019302-2c92-448c-b05f-60f42c531a95-dshm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:29.625376 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.625336 2585 generic.go:356] "Generic (PLEG): container finished" podID="67019302-2c92-448c-b05f-60f42c531a95" containerID="58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24" exitCode=0 Sep 19 12:53:29.625570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.625382 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" Sep 19 12:53:29.625570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.625391 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" event={"ID":"67019302-2c92-448c-b05f-60f42c531a95","Type":"ContainerDied","Data":"58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24"} Sep 19 12:53:29.625570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.625432 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq" event={"ID":"67019302-2c92-448c-b05f-60f42c531a95","Type":"ContainerDied","Data":"74ec750d23218f25a38c1f2d7cc91e9dc8cb9c4a1475b18d5fa3efc730076640"} Sep 19 12:53:29.625570 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.625453 2585 scope.go:117] "RemoveContainer" containerID="58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24" Sep 19 12:53:29.636265 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.636243 2585 scope.go:117] "RemoveContainer" containerID="58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24" Sep 19 12:53:29.636623 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:29.636595 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24\": container with ID starting with 58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24 not found: ID does not exist" containerID="58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24" Sep 19 12:53:29.636724 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.636632 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24"} err="failed to get container status \"58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24\": rpc error: code = NotFound desc = could not find container \"58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24\": container with ID starting with 58993c6b42c3a911722727b60b453581be97ac87a55d5e1c00b22c7f595fef24 not found: ID does not exist" Sep 19 12:53:29.650667 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.650625 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq"] Sep 19 12:53:29.658420 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:29.658386 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d6124/auth-test-model-5d6124-kserve-86f5799f47-nnlzq"] Sep 19 12:53:30.583866 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:30.583820 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67019302-2c92-448c-b05f-60f42c531a95" path="/var/lib/kubelet/pods/67019302-2c92-448c-b05f-60f42c531a95/volumes" Sep 19 12:53:40.191011 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.190974 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9"] Sep 19 12:53:40.191454 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.191214 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" podUID="489f698e-3440-4f04-be22-841180eefc02" containerName="payload-processing" containerID="cri-o://9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6" gracePeriod=30 Sep 19 12:53:40.203272 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.203237 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn"] Sep 19 12:53:40.203520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.203492 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" podUID="4ff3827c-b37f-4acd-b2bb-a886730f2897" containerName="payload-pre-processing" containerID="cri-o://388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955" gracePeriod=30 Sep 19 12:53:40.480594 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.480566 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:53:40.511062 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.511029 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:53:40.524053 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.524019 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8gwf\" (UniqueName: \"kubernetes.io/projected/489f698e-3440-4f04-be22-841180eefc02-kube-api-access-z8gwf\") pod \"489f698e-3440-4f04-be22-841180eefc02\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " Sep 19 12:53:40.524245 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.524217 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f698e-3440-4f04-be22-841180eefc02-plugins-config-volume\") pod \"489f698e-3440-4f04-be22-841180eefc02\" (UID: \"489f698e-3440-4f04-be22-841180eefc02\") " Sep 19 12:53:40.524792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.524762 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489f698e-3440-4f04-be22-841180eefc02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "489f698e-3440-4f04-be22-841180eefc02" (UID: "489f698e-3440-4f04-be22-841180eefc02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:53:40.526964 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.526941 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489f698e-3440-4f04-be22-841180eefc02-kube-api-access-z8gwf" (OuterVolumeSpecName: "kube-api-access-z8gwf") pod "489f698e-3440-4f04-be22-841180eefc02" (UID: "489f698e-3440-4f04-be22-841180eefc02"). InnerVolumeSpecName "kube-api-access-z8gwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:40.625101 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.625063 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff3827c-b37f-4acd-b2bb-a886730f2897-plugins-config-volume\") pod \"4ff3827c-b37f-4acd-b2bb-a886730f2897\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " Sep 19 12:53:40.625307 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.625179 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvfrl\" (UniqueName: \"kubernetes.io/projected/4ff3827c-b37f-4acd-b2bb-a886730f2897-kube-api-access-xvfrl\") pod \"4ff3827c-b37f-4acd-b2bb-a886730f2897\" (UID: \"4ff3827c-b37f-4acd-b2bb-a886730f2897\") " Sep 19 12:53:40.625402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.625372 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f698e-3440-4f04-be22-841180eefc02-plugins-config-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:40.625402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.625391 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gwf\" (UniqueName: \"kubernetes.io/projected/489f698e-3440-4f04-be22-841180eefc02-kube-api-access-z8gwf\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:40.625560 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.625535 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff3827c-b37f-4acd-b2bb-a886730f2897-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ff3827c-b37f-4acd-b2bb-a886730f2897" (UID: "4ff3827c-b37f-4acd-b2bb-a886730f2897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:53:40.627434 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.627412 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff3827c-b37f-4acd-b2bb-a886730f2897-kube-api-access-xvfrl" (OuterVolumeSpecName: "kube-api-access-xvfrl") pod "4ff3827c-b37f-4acd-b2bb-a886730f2897" (UID: "4ff3827c-b37f-4acd-b2bb-a886730f2897"). InnerVolumeSpecName "kube-api-access-xvfrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:40.716096 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.715998 2585 generic.go:356] "Generic (PLEG): container finished" podID="4ff3827c-b37f-4acd-b2bb-a886730f2897" containerID="388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955" exitCode=0 Sep 19 12:53:40.716096 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.716039 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" Sep 19 12:53:40.716096 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.716082 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" event={"ID":"4ff3827c-b37f-4acd-b2bb-a886730f2897","Type":"ContainerDied","Data":"388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955"} Sep 19 12:53:40.716367 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.716134 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn" event={"ID":"4ff3827c-b37f-4acd-b2bb-a886730f2897","Type":"ContainerDied","Data":"e4e4640922d724fe272b280032adaf9b40441a08b5fab7874754402e7b521756"} Sep 19 12:53:40.716367 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.716156 2585 scope.go:117] "RemoveContainer" containerID="388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955" Sep 19 12:53:40.717669 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.717643 2585 generic.go:356] "Generic (PLEG): container finished" podID="489f698e-3440-4f04-be22-841180eefc02" containerID="9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6" exitCode=0 Sep 19 12:53:40.717814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.717718 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" event={"ID":"489f698e-3440-4f04-be22-841180eefc02","Type":"ContainerDied","Data":"9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6"} Sep 19 12:53:40.717814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.717751 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" event={"ID":"489f698e-3440-4f04-be22-841180eefc02","Type":"ContainerDied","Data":"39fe7272cd82a38596f47bfc1c95661d55868add71ab87c304e075f1feaba2e6"} Sep 19 12:53:40.717814 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.717719 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9" Sep 19 12:53:40.726424 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.726403 2585 scope.go:117] "RemoveContainer" containerID="388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955" Sep 19 12:53:40.726595 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.726575 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfrl\" (UniqueName: \"kubernetes.io/projected/4ff3827c-b37f-4acd-b2bb-a886730f2897-kube-api-access-xvfrl\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:40.726646 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.726602 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ff3827c-b37f-4acd-b2bb-a886730f2897-plugins-config-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:40.726788 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:40.726770 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955\": container with ID starting with 388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955 not found: ID does not exist" containerID="388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955" Sep 19 12:53:40.726835 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.726796 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955"} err="failed to get container status \"388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955\": rpc error: code = NotFound desc = could not find container \"388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955\": container with ID starting with 388e157af9c102b78e7e81ddf2ad032f7d4a9ead04f50d107c41b228a127a955 not found: ID does not exist" Sep 19 12:53:40.726835 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.726814 2585 scope.go:117] "RemoveContainer" containerID="9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6" Sep 19 12:53:40.736044 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.736020 2585 scope.go:117] "RemoveContainer" containerID="9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6" Sep 19 12:53:40.736452 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:40.736424 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6\": container with ID starting with 9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6 not found: ID does not exist" containerID="9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6" Sep 19 12:53:40.736549 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.736461 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6"} err="failed to get container status \"9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6\": rpc error: code = NotFound desc = could not find container \"9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6\": container with ID starting with 9ab4ac83a6999b9ef9a033971380839236fa1af4f587fa29a07275975b40dac6 not found: ID does not exist" Sep 19 12:53:40.764191 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.764148 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9"] Sep 19 12:53:40.767231 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.767204 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-04a668-7bf6696589-knqm9"] Sep 19 12:53:40.799711 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.799657 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn"] Sep 19 12:53:40.812909 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:40.812876 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-04a668-c985bd6f8-4zbzn"] Sep 19 12:53:42.581529 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:42.581486 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489f698e-3440-4f04-be22-841180eefc02" path="/var/lib/kubelet/pods/489f698e-3440-4f04-be22-841180eefc02/volumes" Sep 19 12:53:42.582080 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:42.582025 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff3827c-b37f-4acd-b2bb-a886730f2897" path="/var/lib/kubelet/pods/4ff3827c-b37f-4acd-b2bb-a886730f2897/volumes" Sep 19 12:53:44.407133 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.407087 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" podUID="a153fee6-a54e-4158-8a19-c981f0d2e54d" containerName="main" containerID="cri-o://cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c" gracePeriod=45 Sep 19 12:53:44.681924 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.681895 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:53:44.749522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.749477 2585 generic.go:356] "Generic (PLEG): container finished" podID="a153fee6-a54e-4158-8a19-c981f0d2e54d" containerID="cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c" exitCode=0 Sep 19 12:53:44.749522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.749526 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" Sep 19 12:53:44.749848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.749558 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" event={"ID":"a153fee6-a54e-4158-8a19-c981f0d2e54d","Type":"ContainerDied","Data":"cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c"} Sep 19 12:53:44.749848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.749619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx" event={"ID":"a153fee6-a54e-4158-8a19-c981f0d2e54d","Type":"ContainerDied","Data":"8bdc802ae2cc05534c76d95f54d750a22b646a1b140d737685128efdcff376c3"} Sep 19 12:53:44.749848 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.749642 2585 scope.go:117] "RemoveContainer" containerID="cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c" Sep 19 12:53:44.760677 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.760657 2585 scope.go:117] "RemoveContainer" containerID="cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c" Sep 19 12:53:44.761109 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:44.761089 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c\": container with ID starting with cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c not found: ID does not exist" containerID="cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c" Sep 19 12:53:44.761183 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.761122 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c"} err="failed to get container status \"cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c\": rpc error: code = NotFound desc = could not find container \"cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c\": container with ID starting with cf2c26fd3025af9acc73ce468ed4bd3e46f2dba7b0e520e8841b5ef0bc5e445c not found: ID does not exist" Sep 19 12:53:44.767517 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767494 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gsnm\" (UniqueName: \"kubernetes.io/projected/a153fee6-a54e-4158-8a19-c981f0d2e54d-kube-api-access-5gsnm\") pod \"a153fee6-a54e-4158-8a19-c981f0d2e54d\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " Sep 19 12:53:44.767628 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767531 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-home\") pod \"a153fee6-a54e-4158-8a19-c981f0d2e54d\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " Sep 19 12:53:44.767628 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767550 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a153fee6-a54e-4158-8a19-c981f0d2e54d-tls-certs\") pod \"a153fee6-a54e-4158-8a19-c981f0d2e54d\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " Sep 19 12:53:44.767628 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767573 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-model-cache\") pod \"a153fee6-a54e-4158-8a19-c981f0d2e54d\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " Sep 19 12:53:44.767818 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767632 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-tmp-dir\") pod \"a153fee6-a54e-4158-8a19-c981f0d2e54d\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " Sep 19 12:53:44.767915 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767867 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-home" (OuterVolumeSpecName: "home") pod "a153fee6-a54e-4158-8a19-c981f0d2e54d" (UID: "a153fee6-a54e-4158-8a19-c981f0d2e54d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:44.768058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767914 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a153fee6-a54e-4158-8a19-c981f0d2e54d" (UID: "a153fee6-a54e-4158-8a19-c981f0d2e54d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:44.768058 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.767961 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a153fee6-a54e-4158-8a19-c981f0d2e54d" (UID: "a153fee6-a54e-4158-8a19-c981f0d2e54d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:44.768276 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.768237 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-dshm\") pod \"a153fee6-a54e-4158-8a19-c981f0d2e54d\" (UID: \"a153fee6-a54e-4158-8a19-c981f0d2e54d\") " Sep 19 12:53:44.769139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.768613 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-home\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:44.769139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.768636 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-model-cache\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:44.769139 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.768678 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-tmp-dir\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:44.771430 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.771391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-dshm" (OuterVolumeSpecName: "dshm") pod "a153fee6-a54e-4158-8a19-c981f0d2e54d" (UID: "a153fee6-a54e-4158-8a19-c981f0d2e54d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:44.771430 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.771418 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a153fee6-a54e-4158-8a19-c981f0d2e54d-kube-api-access-5gsnm" (OuterVolumeSpecName: "kube-api-access-5gsnm") pod "a153fee6-a54e-4158-8a19-c981f0d2e54d" (UID: "a153fee6-a54e-4158-8a19-c981f0d2e54d"). InnerVolumeSpecName "kube-api-access-5gsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:44.772484 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.772456 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a153fee6-a54e-4158-8a19-c981f0d2e54d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a153fee6-a54e-4158-8a19-c981f0d2e54d" (UID: "a153fee6-a54e-4158-8a19-c981f0d2e54d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:53:44.869945 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.869904 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsnm\" (UniqueName: \"kubernetes.io/projected/a153fee6-a54e-4158-8a19-c981f0d2e54d-kube-api-access-5gsnm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:44.869945 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.869948 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a153fee6-a54e-4158-8a19-c981f0d2e54d-tls-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:44.870187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:44.869962 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a153fee6-a54e-4158-8a19-c981f0d2e54d-dshm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:45.075093 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:45.075055 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx"] Sep 19 12:53:45.080627 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:45.080593 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-04a668/premium-w4-04a668-kserve-7565dbb78c-bvsrx"] Sep 19 12:53:46.581450 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:46.581418 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a153fee6-a54e-4158-8a19-c981f0d2e54d" path="/var/lib/kubelet/pods/a153fee6-a54e-4158-8a19-c981f0d2e54d/volumes" Sep 19 12:53:47.162937 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:47.162901 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v"] Sep 19 12:53:47.163176 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:47.163150 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" containerID="cri-o://d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3" gracePeriod=30 Sep 19 12:53:52.433931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.433861 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:53:52.552499 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552470 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-podinfo\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552499 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552505 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-token\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552545 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-data\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552627 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-socket\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552678 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6af4c68-7371-4196-a2eb-961dbf2229a5-istiod-ca-cert\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552765 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552730 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-certs\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552787 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkr5j\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-kube-api-access-kkr5j\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.552982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552934 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:52.552982 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.552948 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:52.553143 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553034 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:52.553217 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553196 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-credential-socket\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.553217 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553205 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:52.553325 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553231 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6af4c68-7371-4196-a2eb-961dbf2229a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:53:52.553325 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553239 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-envoy\") pod \"d6af4c68-7371-4196-a2eb-961dbf2229a5\" (UID: \"d6af4c68-7371-4196-a2eb-961dbf2229a5\") " Sep 19 12:53:52.553584 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553564 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.553652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553590 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6af4c68-7371-4196-a2eb-961dbf2229a5-istiod-ca-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.553652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553603 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-workload-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.553652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553615 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-credential-socket\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.553652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.553628 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-data\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.554916 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.554895 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:52.555288 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.555270 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:53:52.555404 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.555381 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 12:53:52.555492 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.555401 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-kube-api-access-kkr5j" (OuterVolumeSpecName: "kube-api-access-kkr5j") pod "d6af4c68-7371-4196-a2eb-961dbf2229a5" (UID: "d6af4c68-7371-4196-a2eb-961dbf2229a5"). InnerVolumeSpecName "kube-api-access-kkr5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:53:52.654863 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.654817 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkr5j\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-kube-api-access-kkr5j\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.654863 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.654860 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-envoy\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.655125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.654877 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-podinfo\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.655125 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.654889 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6af4c68-7371-4196-a2eb-961dbf2229a5-istio-token\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:53:52.807030 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.806999 2585 generic.go:356] "Generic (PLEG): container finished" podID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerID="d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3" exitCode=0 Sep 19 12:53:52.807210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.807041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" event={"ID":"d6af4c68-7371-4196-a2eb-961dbf2229a5","Type":"ContainerDied","Data":"d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3"} Sep 19 12:53:52.807210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.807057 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" Sep 19 12:53:52.807210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.807100 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v" event={"ID":"d6af4c68-7371-4196-a2eb-961dbf2229a5","Type":"ContainerDied","Data":"6d2b012efcfdecf7123fd247a22db7642c1bf4f3feac27369724a0b3cff5251d"} Sep 19 12:53:52.807210 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.807124 2585 scope.go:117] "RemoveContainer" containerID="d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3" Sep 19 12:53:52.817088 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.817068 2585 scope.go:117] "RemoveContainer" containerID="dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d" Sep 19 12:53:52.826823 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.826802 2585 scope.go:117] "RemoveContainer" containerID="d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3" Sep 19 12:53:52.826926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.826881 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v"] Sep 19 12:53:52.827152 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:52.827129 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3\": container with ID starting with d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3 not found: ID does not exist" containerID="d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3" Sep 19 12:53:52.827262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.827158 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3"} err="failed to get container status \"d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3\": rpc error: code = NotFound desc = could not find container \"d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3\": container with ID starting with d8bf9f34dacf442011969d53daff4cf535879e906cb0fd49ea5343860325f9d3 not found: ID does not exist" Sep 19 12:53:52.827262 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.827178 2585 scope.go:117] "RemoveContainer" containerID="dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d" Sep 19 12:53:52.827450 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:53:52.827431 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d\": container with ID starting with dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d not found: ID does not exist" containerID="dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d" Sep 19 12:53:52.827493 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.827458 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d"} err="failed to get container status \"dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d\": rpc error: code = NotFound desc = could not find container \"dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d\": container with ID starting with dc72c6879c70d4b24a039c261918d0673256c7f3ae3abca7c82dbf9d9e58b38d not found: ID does not exist" Sep 19 12:53:52.831538 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:52.831515 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-04a668-openshift-default-555fc97686-sf64v"] Sep 19 12:53:54.588203 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:53:54.588168 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" path="/var/lib/kubelet/pods/d6af4c68-7371-4196-a2eb-961dbf2229a5/volumes" Sep 19 12:54:46.676463 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676376 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-0d5a6f"] Sep 19 12:54:46.676919 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676903 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489f698e-3440-4f04-be22-841180eefc02" containerName="payload-processing" Sep 19 12:54:46.676966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676922 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="489f698e-3440-4f04-be22-841180eefc02" containerName="payload-processing" Sep 19 12:54:46.676966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676930 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" containerName="test-curl-7567-04161d" Sep 19 12:54:46.676966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676936 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" containerName="test-curl-7567-04161d" Sep 19 12:54:46.676966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676950 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67019302-2c92-448c-b05f-60f42c531a95" containerName="main" Sep 19 12:54:46.676966 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676955 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="67019302-2c92-448c-b05f-60f42c531a95" containerName="main" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676968 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676976 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676984 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676989 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.676997 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff3827c-b37f-4acd-b2bb-a886730f2897" containerName="payload-pre-processing" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677003 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff3827c-b37f-4acd-b2bb-a886730f2897" containerName="payload-pre-processing" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677019 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a153fee6-a54e-4158-8a19-c981f0d2e54d" containerName="main" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677024 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a153fee6-a54e-4158-8a19-c981f0d2e54d" containerName="main" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677096 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a153fee6-a54e-4158-8a19-c981f0d2e54d" containerName="main" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677104 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="64dbb6f1-1a19-4b10-8579-14a7a20ad562" containerName="test-curl-7567-04161d" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677110 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="489f698e-3440-4f04-be22-841180eefc02" containerName="payload-processing" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677115 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="67019302-2c92-448c-b05f-60f42c531a95" containerName="main" Sep 19 12:54:46.677123 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677124 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd27e7d2-fe3c-4ee5-a3aa-b0866e8e1687" containerName="istio-proxy" Sep 19 12:54:46.677520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677133 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" Sep 19 12:54:46.677520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677141 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff3827c-b37f-4acd-b2bb-a886730f2897" containerName="payload-pre-processing" Sep 19 12:54:46.677520 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.677148 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6af4c68-7371-4196-a2eb-961dbf2229a5" containerName="istio-proxy" Sep 19 12:54:46.680531 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.680510 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:46.683377 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.683352 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pjdxd\"" Sep 19 12:54:46.686912 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.686883 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-0d5a6f"] Sep 19 12:54:46.704306 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.704274 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8c2v\" (UniqueName: \"kubernetes.io/projected/de3bc089-b9a9-41a0-903c-0d46aac4eab4-kube-api-access-k8c2v\") pod \"test-curl-7567-0d5a6f\" (UID: \"de3bc089-b9a9-41a0-903c-0d46aac4eab4\") " pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:46.805652 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.805614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8c2v\" (UniqueName: \"kubernetes.io/projected/de3bc089-b9a9-41a0-903c-0d46aac4eab4-kube-api-access-k8c2v\") pod \"test-curl-7567-0d5a6f\" (UID: \"de3bc089-b9a9-41a0-903c-0d46aac4eab4\") " pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:46.813964 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.813939 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8c2v\" (UniqueName: \"kubernetes.io/projected/de3bc089-b9a9-41a0-903c-0d46aac4eab4-kube-api-access-k8c2v\") pod \"test-curl-7567-0d5a6f\" (UID: \"de3bc089-b9a9-41a0-903c-0d46aac4eab4\") " pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:46.993717 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:46.993595 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:47.271734 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:47.271705 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-0d5a6f"] Sep 19 12:54:47.272313 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:54:47.272287 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde3bc089_b9a9_41a0_903c_0d46aac4eab4.slice/crio-cb8832c2675f2658c51b2ca7c45834cad81f21dd51916ddf8b0c31c4b4570930 WatchSource:0}: Error finding container cb8832c2675f2658c51b2ca7c45834cad81f21dd51916ddf8b0c31c4b4570930: Status 404 returned error can't find the container with id cb8832c2675f2658c51b2ca7c45834cad81f21dd51916ddf8b0c31c4b4570930 Sep 19 12:54:48.175948 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.175911 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-0d5a6f" event={"ID":"de3bc089-b9a9-41a0-903c-0d46aac4eab4","Type":"ContainerStarted","Data":"3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4"} Sep 19 12:54:48.176387 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.175962 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-0d5a6f" event={"ID":"de3bc089-b9a9-41a0-903c-0d46aac4eab4","Type":"ContainerStarted","Data":"cb8832c2675f2658c51b2ca7c45834cad81f21dd51916ddf8b0c31c4b4570930"} Sep 19 12:54:48.192934 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.192858 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7567-0d5a6f" podStartSLOduration=2.192841257 podStartE2EDuration="2.192841257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:54:48.189313668 +0000 UTC m=+1783.796164246" watchObservedRunningTime="2026-09-19 12:54:48.192841257 +0000 UTC m=+1783.799691835" Sep 19 12:54:48.755861 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.755822 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-0d5a6f"] Sep 19 12:54:48.758843 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.758811 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-0d5a6f"] Sep 19 12:54:48.871024 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.870936 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-e8241e"] Sep 19 12:54:48.871726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.871707 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" containerName="test-curl-7567-0d5a6f" Sep 19 12:54:48.871779 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.871728 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" containerName="test-curl-7567-0d5a6f" Sep 19 12:54:48.871873 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.871862 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" containerName="test-curl-7567-0d5a6f" Sep 19 12:54:48.875721 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.875676 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:48.878006 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.877982 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-e8241e"] Sep 19 12:54:48.925363 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:48.925333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b79tq\" (UniqueName: \"kubernetes.io/projected/623be05b-6c6d-4c2a-a0bf-fee57bf662ba-kube-api-access-b79tq\") pod \"test-curl-7567-e8241e\" (UID: \"623be05b-6c6d-4c2a-a0bf-fee57bf662ba\") " pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:49.026275 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:49.026242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b79tq\" (UniqueName: \"kubernetes.io/projected/623be05b-6c6d-4c2a-a0bf-fee57bf662ba-kube-api-access-b79tq\") pod \"test-curl-7567-e8241e\" (UID: \"623be05b-6c6d-4c2a-a0bf-fee57bf662ba\") " pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:49.034588 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:49.034538 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79tq\" (UniqueName: \"kubernetes.io/projected/623be05b-6c6d-4c2a-a0bf-fee57bf662ba-kube-api-access-b79tq\") pod \"test-curl-7567-e8241e\" (UID: \"623be05b-6c6d-4c2a-a0bf-fee57bf662ba\") " pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:49.182933 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:49.182844 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7567-0d5a6f" container="test-curl-7567-0d5a6f" Sep 19 12:54:49.187149 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:49.187122 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:49.468427 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:49.462101 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-e8241e"] Sep 19 12:54:49.471666 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:54:49.471626 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod623be05b_6c6d_4c2a_a0bf_fee57bf662ba.slice/crio-bb2c2f782ec7903d05f28b1e9e72761221b5f83beade71ae5a36b4355bb42654 WatchSource:0}: Error finding container bb2c2f782ec7903d05f28b1e9e72761221b5f83beade71ae5a36b4355bb42654: Status 404 returned error can't find the container with id bb2c2f782ec7903d05f28b1e9e72761221b5f83beade71ae5a36b4355bb42654 Sep 19 12:54:50.190846 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:50.190812 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-e8241e" event={"ID":"623be05b-6c6d-4c2a-a0bf-fee57bf662ba","Type":"ContainerStarted","Data":"af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905"} Sep 19 12:54:50.191333 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:50.190856 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-e8241e" event={"ID":"623be05b-6c6d-4c2a-a0bf-fee57bf662ba","Type":"ContainerStarted","Data":"bb2c2f782ec7903d05f28b1e9e72761221b5f83beade71ae5a36b4355bb42654"} Sep 19 12:54:50.191333 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:50.191101 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7567-0d5a6f" podUID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" containerName="test-curl-7567-0d5a6f" containerID="cri-o://3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4" gracePeriod=2 Sep 19 12:54:50.206438 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:50.206403 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-0d5a6f\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:50.207505 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:50.207463 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7567-e8241e" podStartSLOduration=2.207449289 podStartE2EDuration="2.207449289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:54:50.204605066 +0000 UTC m=+1785.811455666" watchObservedRunningTime="2026-09-19 12:54:50.207449289 +0000 UTC m=+1785.814299868" Sep 19 12:54:50.789640 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:50.789604 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-e8241e"] Sep 19 12:54:50.793563 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:50.793532 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-e8241e"] Sep 19 12:54:51.198346 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:51.198310 2585 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7567-e8241e" container="test-curl-7567-e8241e" Sep 19 12:54:52.206748 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:52.206706 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7567-e8241e" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" containerName="test-curl-7567-e8241e" containerID="cri-o://af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905" gracePeriod=2 Sep 19 12:54:52.209468 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:52.209422 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-e8241e\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:52.370316 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:52.370282 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:52.372538 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:52.372507 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-e8241e\" is forbidden: User \"system:node:ip-10-0-131-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-149.ec2.internal' and this object" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:52.460365 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:52.460264 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8c2v\" (UniqueName: \"kubernetes.io/projected/de3bc089-b9a9-41a0-903c-0d46aac4eab4-kube-api-access-k8c2v\") pod \"de3bc089-b9a9-41a0-903c-0d46aac4eab4\" (UID: \"de3bc089-b9a9-41a0-903c-0d46aac4eab4\") " Sep 19 12:54:52.462610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:52.462584 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3bc089-b9a9-41a0-903c-0d46aac4eab4-kube-api-access-k8c2v" (OuterVolumeSpecName: "kube-api-access-k8c2v") pod "de3bc089-b9a9-41a0-903c-0d46aac4eab4" (UID: "de3bc089-b9a9-41a0-903c-0d46aac4eab4"). InnerVolumeSpecName "kube-api-access-k8c2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:54:52.561303 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:52.561253 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8c2v\" (UniqueName: \"kubernetes.io/projected/de3bc089-b9a9-41a0-903c-0d46aac4eab4-kube-api-access-k8c2v\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:54:52.579755 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:52.579722 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" path="/var/lib/kubelet/pods/de3bc089-b9a9-41a0-903c-0d46aac4eab4/volumes" Sep 19 12:54:53.214419 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:53.214382 2585 generic.go:356] "Generic (PLEG): container finished" podID="de3bc089-b9a9-41a0-903c-0d46aac4eab4" containerID="3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4" exitCode=137 Sep 19 12:54:53.214939 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:53.214430 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-0d5a6f" Sep 19 12:54:53.214939 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:53.214489 2585 scope.go:117] "RemoveContainer" containerID="3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4" Sep 19 12:54:53.224112 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:53.224090 2585 scope.go:117] "RemoveContainer" containerID="3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4" Sep 19 12:54:53.224421 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:53.224400 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4\": container with ID starting with 3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4 not found: ID does not exist" containerID="3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4" Sep 19 12:54:53.224467 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:53.224431 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4"} err="failed to get container status \"3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4\": rpc error: code = NotFound desc = could not find container \"3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4\": container with ID starting with 3c8d947b1feff05d5f212ab244de548ec2dba8db035cc539b15fd1791a51b0d4 not found: ID does not exist" Sep 19 12:54:54.411697 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:54.411668 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:54.479069 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:54.478973 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b79tq\" (UniqueName: \"kubernetes.io/projected/623be05b-6c6d-4c2a-a0bf-fee57bf662ba-kube-api-access-b79tq\") pod \"623be05b-6c6d-4c2a-a0bf-fee57bf662ba\" (UID: \"623be05b-6c6d-4c2a-a0bf-fee57bf662ba\") " Sep 19 12:54:54.481208 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:54.481177 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623be05b-6c6d-4c2a-a0bf-fee57bf662ba-kube-api-access-b79tq" (OuterVolumeSpecName: "kube-api-access-b79tq") pod "623be05b-6c6d-4c2a-a0bf-fee57bf662ba" (UID: "623be05b-6c6d-4c2a-a0bf-fee57bf662ba"). InnerVolumeSpecName "kube-api-access-b79tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:54:54.584777 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:54.584738 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b79tq\" (UniqueName: \"kubernetes.io/projected/623be05b-6c6d-4c2a-a0bf-fee57bf662ba-kube-api-access-b79tq\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:54:54.587048 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:54.587019 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" path="/var/lib/kubelet/pods/623be05b-6c6d-4c2a-a0bf-fee57bf662ba/volumes" Sep 19 12:54:55.231792 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:55.231754 2585 generic.go:356] "Generic (PLEG): container finished" podID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" containerID="af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905" exitCode=137 Sep 19 12:54:55.231985 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:55.231801 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e8241e" Sep 19 12:54:55.231985 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:55.231846 2585 scope.go:117] "RemoveContainer" containerID="af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905" Sep 19 12:54:55.241143 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:55.241122 2585 scope.go:117] "RemoveContainer" containerID="af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905" Sep 19 12:54:55.241465 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:54:55.241435 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905\": container with ID starting with af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905 not found: ID does not exist" containerID="af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905" Sep 19 12:54:55.241527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:55.241474 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905"} err="failed to get container status \"af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905\": rpc error: code = NotFound desc = could not find container \"af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905\": container with ID starting with af90cd8ab11821b7f47540de579f475f21a75e0b4120796ec080f3460100e905 not found: ID does not exist" Sep 19 12:54:59.340218 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.340181 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb"] Sep 19 12:54:59.341061 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.341032 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" containerName="test-curl-7567-e8241e" Sep 19 12:54:59.341061 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.341062 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" containerName="test-curl-7567-e8241e" Sep 19 12:54:59.341276 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.341167 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="623be05b-6c6d-4c2a-a0bf-fee57bf662ba" containerName="test-curl-7567-e8241e" Sep 19 12:54:59.344830 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.344810 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.347370 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.347346 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b0e30e\"/\"openshift-service-ca.crt\"" Sep 19 12:54:59.348212 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.348189 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b0e30e\"/\"kube-root-ca.crt\"" Sep 19 12:54:59.348326 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.348252 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b0e30e\"/\"rate-test-model-b0e30e-kserve-self-signed-certs\"" Sep 19 12:54:59.348326 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.348318 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b0e30e\"/\"default-dockercfg-gmwjx\"" Sep 19 12:54:59.355081 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.355054 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb"] Sep 19 12:54:59.431971 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.431919 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-home\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.431971 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.431976 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-dshm\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.432225 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.432007 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tls-certs\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.432225 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.432024 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tmp-dir\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.432225 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.432044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-model-cache\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.432225 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.432066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpn9\" (UniqueName: \"kubernetes.io/projected/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-kube-api-access-kfpn9\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.533527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.533486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-home\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.533527 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.533531 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-dshm\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.533913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.533557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tls-certs\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.533913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.533582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tmp-dir\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.533913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.533608 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-model-cache\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.533913 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.533648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpn9\" (UniqueName: \"kubernetes.io/projected/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-kube-api-access-kfpn9\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.534119 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.534000 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-home\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.534119 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.534063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tmp-dir\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.534119 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.534105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-model-cache\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.536151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.536128 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-dshm\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.536432 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.536413 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tls-certs\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.541834 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.541811 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpn9\" (UniqueName: \"kubernetes.io/projected/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-kube-api-access-kfpn9\") pod \"rate-test-model-b0e30e-kserve-6b8799b95f-b9frb\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.656726 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.656589 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:54:59.933911 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:54:59.933873 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb"] Sep 19 12:54:59.935235 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:54:59.935200 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79a86e3b_ddd7_4f92_84be_8d0fb96f4e90.slice/crio-0c92cc18789e4b1c1726329bb7b8bcfaa56bb57bcf473899d142b94b0e11e917 WatchSource:0}: Error finding container 0c92cc18789e4b1c1726329bb7b8bcfaa56bb57bcf473899d142b94b0e11e917: Status 404 returned error can't find the container with id 0c92cc18789e4b1c1726329bb7b8bcfaa56bb57bcf473899d142b94b0e11e917 Sep 19 12:55:00.267821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:00.267668 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" event={"ID":"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90","Type":"ContainerStarted","Data":"d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a"} Sep 19 12:55:00.267821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:00.267770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" event={"ID":"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90","Type":"ContainerStarted","Data":"0c92cc18789e4b1c1726329bb7b8bcfaa56bb57bcf473899d142b94b0e11e917"} Sep 19 12:55:00.287069 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:00.287014 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" podStartSLOduration=1.286995615 podStartE2EDuration="1.286995615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:55:00.28474598 +0000 UTC m=+1795.891596770" watchObservedRunningTime="2026-09-19 12:55:00.286995615 +0000 UTC m=+1795.893846193" Sep 19 12:55:06.699621 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:06.699592 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:55:06.702113 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:06.702092 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 12:55:09.657395 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:09.657355 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:55:09.657914 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:09.657427 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:55:09.671926 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:09.671892 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:55:10.344645 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:55:10.344614 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:56:10.717710 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:10.717663 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb"] Sep 19 12:56:17.126531 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.126457 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499"] Sep 19 12:56:17.130695 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.130655 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.136782 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.136761 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f32129-serving-cert\"" Sep 19 12:56:17.148772 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.148749 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499"] Sep 19 12:56:17.206603 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.206571 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqvv\" (UniqueName: \"kubernetes.io/projected/96fd1262-763b-45d2-b285-492aca342d42-kube-api-access-plqvv\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.206777 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.206710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fd1262-763b-45d2-b285-492aca342d42-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.206777 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.206740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-metrics-certs\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.206777 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.206767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.308151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.308121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fd1262-763b-45d2-b285-492aca342d42-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.308151 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.308151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-metrics-certs\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.308360 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.308176 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.308360 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.308200 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqvv\" (UniqueName: \"kubernetes.io/projected/96fd1262-763b-45d2-b285-492aca342d42-kube-api-access-plqvv\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.308876 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.308848 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fd1262-763b-45d2-b285-492aca342d42-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.311031 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.311008 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-metrics-certs\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.311118 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.311042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.322285 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.322265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqvv\" (UniqueName: \"kubernetes.io/projected/96fd1262-763b-45d2-b285-492aca342d42-kube-api-access-plqvv\") pod \"maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.443976 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.443897 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:17.722644 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.722613 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499"] Sep 19 12:56:17.724311 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:56:17.724278 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fd1262_763b_45d2_b285_492aca342d42.slice/crio-045817651a5baa165934ae5b07d384e7f12b92b1c03d822504f627330cfbb01c WatchSource:0}: Error finding container 045817651a5baa165934ae5b07d384e7f12b92b1c03d822504f627330cfbb01c: Status 404 returned error can't find the container with id 045817651a5baa165934ae5b07d384e7f12b92b1c03d822504f627330cfbb01c Sep 19 12:56:17.725963 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.725938 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 12:56:17.775261 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:17.775226 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" event={"ID":"96fd1262-763b-45d2-b285-492aca342d42","Type":"ContainerStarted","Data":"045817651a5baa165934ae5b07d384e7f12b92b1c03d822504f627330cfbb01c"} Sep 19 12:56:18.784256 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:18.784168 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" event={"ID":"96fd1262-763b-45d2-b285-492aca342d42","Type":"ContainerStarted","Data":"aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23"} Sep 19 12:56:18.784256 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:18.784226 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:18.801904 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:18.801860 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" podStartSLOduration=1.194262464 podStartE2EDuration="1.801845365s" totalImagesPullingTime="607.582901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:56:18.80014329 +0000 UTC m=+1874.406993896" watchObservedRunningTime="2026-09-19 12:56:18.801845365 +0000 UTC m=+1874.408695944" Sep 19 12:56:24.797550 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:24.797524 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:56:25.738664 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:25.738617 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" podUID="79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" containerName="main" containerID="cri-o://d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a" gracePeriod=45 Sep 19 12:56:26.015994 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.015972 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:56:26.088673 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088646 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-home\") pod \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " Sep 19 12:56:26.088821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088711 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tls-certs\") pod \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " Sep 19 12:56:26.088821 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088757 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tmp-dir\") pod \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " Sep 19 12:56:26.088961 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088940 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-model-cache\") pod \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " Sep 19 12:56:26.089017 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088977 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-dshm\") pod \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " Sep 19 12:56:26.089017 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088996 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfpn9\" (UniqueName: \"kubernetes.io/projected/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-kube-api-access-kfpn9\") pod \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\" (UID: \"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90\") " Sep 19 12:56:26.089017 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088928 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-home" (OuterVolumeSpecName: "home") pod "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" (UID: "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:56:26.089170 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.088992 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" (UID: "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:56:26.089170 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.089102 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-model-cache" (OuterVolumeSpecName: "model-cache") pod "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" (UID: "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:56:26.089402 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.089385 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-model-cache\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:56:26.089475 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.089407 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-home\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:56:26.089475 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.089420 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tmp-dir\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:56:26.091168 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.091150 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-dshm" (OuterVolumeSpecName: "dshm") pod "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" (UID: "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 12:56:26.091233 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.091219 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" (UID: "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:56:26.091270 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.091254 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-kube-api-access-kfpn9" (OuterVolumeSpecName: "kube-api-access-kfpn9") pod "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" (UID: "79a86e3b-ddd7-4f92-84be-8d0fb96f4e90"). InnerVolumeSpecName "kube-api-access-kfpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:56:26.190932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.190905 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-tls-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:56:26.190932 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.190928 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-dshm\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:56:26.191080 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.190939 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfpn9\" (UniqueName: \"kubernetes.io/projected/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90-kube-api-access-kfpn9\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:56:26.837880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.837796 2585 generic.go:356] "Generic (PLEG): container finished" podID="79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" containerID="d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a" exitCode=0 Sep 19 12:56:26.837880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.837830 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" event={"ID":"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90","Type":"ContainerDied","Data":"d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a"} Sep 19 12:56:26.837880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.837835 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" Sep 19 12:56:26.837880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.837856 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb" event={"ID":"79a86e3b-ddd7-4f92-84be-8d0fb96f4e90","Type":"ContainerDied","Data":"0c92cc18789e4b1c1726329bb7b8bcfaa56bb57bcf473899d142b94b0e11e917"} Sep 19 12:56:26.837880 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.837872 2585 scope.go:117] "RemoveContainer" containerID="d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a" Sep 19 12:56:26.849297 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.849278 2585 scope.go:117] "RemoveContainer" containerID="d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a" Sep 19 12:56:26.849586 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:56:26.849567 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a\": container with ID starting with d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a not found: ID does not exist" containerID="d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a" Sep 19 12:56:26.849651 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.849593 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a"} err="failed to get container status \"d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a\": rpc error: code = NotFound desc = could not find container \"d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a\": container with ID starting with d1e9f0203efaf38d4ca9e5821cebaf723a8a9600d39d6c63a1088d818de04f4a not found: ID does not exist" Sep 19 12:56:26.876737 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.876702 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb"] Sep 19 12:56:26.888911 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:26.888874 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b0e30e/rate-test-model-b0e30e-kserve-6b8799b95f-b9frb"] Sep 19 12:56:28.588056 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:28.588023 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" path="/var/lib/kubelet/pods/79a86e3b-ddd7-4f92-84be-8d0fb96f4e90/volumes" Sep 19 12:56:51.740442 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.740409 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5"] Sep 19 12:56:51.740950 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.740907 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" containerName="main" Sep 19 12:56:51.740950 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.740920 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" containerName="main" Sep 19 12:56:51.741037 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.740998 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a86e3b-ddd7-4f92-84be-8d0fb96f4e90" containerName="main" Sep 19 12:56:51.745548 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.745524 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.748036 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.748017 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4e754a-serving-cert\"" Sep 19 12:56:51.756391 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.755748 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5"] Sep 19 12:56:51.822923 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.822892 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.823077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.822942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhj9b\" (UniqueName: \"kubernetes.io/projected/eff00aa4-b618-4325-af81-1400958a2faa-kube-api-access-xhj9b\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.823077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.822992 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff00aa4-b618-4325-af81-1400958a2faa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.823077 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.823018 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-metrics-certs\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.923811 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.923780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff00aa4-b618-4325-af81-1400958a2faa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.923960 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.923816 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-metrics-certs\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.923960 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.923901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.923960 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.923924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhj9b\" (UniqueName: \"kubernetes.io/projected/eff00aa4-b618-4325-af81-1400958a2faa-kube-api-access-xhj9b\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.924070 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:56:51.924015 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-serving-cert: secret "maas-api-e2e-ipp-b-4e754a-serving-cert" not found Sep 19 12:56:51.924110 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:56:51.924079 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls podName:eff00aa4-b618-4325-af81-1400958a2faa nodeName:}" failed. No retries permitted until 2026-09-19 12:56:52.424060998 +0000 UTC m=+1908.030911747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls") pod "maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" (UID: "eff00aa4-b618-4325-af81-1400958a2faa") : secret "maas-api-e2e-ipp-b-4e754a-serving-cert" not found Sep 19 12:56:51.924547 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.924528 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff00aa4-b618-4325-af81-1400958a2faa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.926521 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.926493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-metrics-certs\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:51.932893 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:51.932874 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhj9b\" (UniqueName: \"kubernetes.io/projected/eff00aa4-b618-4325-af81-1400958a2faa-kube-api-access-xhj9b\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:52.430449 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:52.430412 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:52.433048 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:52.433023 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:52.661415 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:52.661385 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:52.941858 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:52.941834 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5"] Sep 19 12:56:52.943672 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:56:52.943646 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff00aa4_b618_4325_af81_1400958a2faa.slice/crio-364fd14cb1109a84a84194a014656254cca28ee24640ec611405c6815d983319 WatchSource:0}: Error finding container 364fd14cb1109a84a84194a014656254cca28ee24640ec611405c6815d983319: Status 404 returned error can't find the container with id 364fd14cb1109a84a84194a014656254cca28ee24640ec611405c6815d983319 Sep 19 12:56:53.011390 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:53.011305 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" event={"ID":"eff00aa4-b618-4325-af81-1400958a2faa","Type":"ContainerStarted","Data":"364fd14cb1109a84a84194a014656254cca28ee24640ec611405c6815d983319"} Sep 19 12:56:54.019357 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:54.019319 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" event={"ID":"eff00aa4-b618-4325-af81-1400958a2faa","Type":"ContainerStarted","Data":"5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88"} Sep 19 12:56:54.019754 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:54.019474 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:56:54.039052 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:56:54.039004 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" podStartSLOduration=2.7400470070000003 podStartE2EDuration="3.038990119s" totalImagesPullingTime="298.943112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:56:54.037142439 +0000 UTC m=+1909.643993039" watchObservedRunningTime="2026-09-19 12:56:54.038990119 +0000 UTC m=+1909.645840697" Sep 19 12:57:00.031984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:57:00.031952 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 12:58:07.846059 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.845974 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4"] Sep 19 12:58:07.850537 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.850513 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:07.852774 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.852751 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1593f0-serving-cert\"" Sep 19 12:58:07.862742 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.862700 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4"] Sep 19 12:58:07.909216 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.909184 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33efb618-140c-42f1-a75c-708eb2de3bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:07.909342 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.909226 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:07.909342 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.909258 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:07.909342 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:07.909273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zlp\" (UniqueName: \"kubernetes.io/projected/33efb618-140c-42f1-a75c-708eb2de3bab-kube-api-access-n7zlp\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.010049 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.010023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33efb618-140c-42f1-a75c-708eb2de3bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.010161 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.010067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.010161 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.010105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.010161 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.010123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zlp\" (UniqueName: \"kubernetes.io/projected/33efb618-140c-42f1-a75c-708eb2de3bab-kube-api-access-n7zlp\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.010766 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.010738 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33efb618-140c-42f1-a75c-708eb2de3bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.012670 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.012642 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.012777 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.012739 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.017916 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.017897 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zlp\" (UniqueName: \"kubernetes.io/projected/33efb618-140c-42f1-a75c-708eb2de3bab-kube-api-access-n7zlp\") pod \"maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.163585 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.163512 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:08.444484 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.444456 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4"] Sep 19 12:58:08.447125 ip-10-0-131-149 kubenswrapper[2585]: W0919 12:58:08.447096 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33efb618_140c_42f1_a75c_708eb2de3bab.slice/crio-ea2b7bc6409a9854c0d0c0622c522c8aff575640853b79cc2bafb41993ae0399 WatchSource:0}: Error finding container ea2b7bc6409a9854c0d0c0622c522c8aff575640853b79cc2bafb41993ae0399: Status 404 returned error can't find the container with id ea2b7bc6409a9854c0d0c0622c522c8aff575640853b79cc2bafb41993ae0399 Sep 19 12:58:08.495312 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:08.495279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" event={"ID":"33efb618-140c-42f1-a75c-708eb2de3bab","Type":"ContainerStarted","Data":"ea2b7bc6409a9854c0d0c0622c522c8aff575640853b79cc2bafb41993ae0399"} Sep 19 12:58:09.503503 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:09.503465 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" event={"ID":"33efb618-140c-42f1-a75c-708eb2de3bab","Type":"ContainerStarted","Data":"472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55"} Sep 19 12:58:09.503908 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:09.503534 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:09.520522 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:09.520476 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" podStartSLOduration=2.125273254 podStartE2EDuration="2.520460421s" totalImagesPullingTime="395.187167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:58:09.51882044 +0000 UTC m=+1985.125671044" watchObservedRunningTime="2026-09-19 12:58:09.520460421 +0000 UTC m=+1985.127311000" Sep 19 12:58:15.515943 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:15.515913 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:46.449452 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.449398 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4"] Sep 19 12:58:46.450050 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.449705 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" podUID="33efb618-140c-42f1-a75c-708eb2de3bab" containerName="maas-api" containerID="cri-o://472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55" gracePeriod=30 Sep 19 12:58:46.715005 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.714936 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:46.731628 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.731600 2585 generic.go:356] "Generic (PLEG): container finished" podID="33efb618-140c-42f1-a75c-708eb2de3bab" containerID="472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55" exitCode=0 Sep 19 12:58:46.731785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.731638 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" event={"ID":"33efb618-140c-42f1-a75c-708eb2de3bab","Type":"ContainerDied","Data":"472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55"} Sep 19 12:58:46.731785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.731646 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" Sep 19 12:58:46.731785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.731716 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4" event={"ID":"33efb618-140c-42f1-a75c-708eb2de3bab","Type":"ContainerDied","Data":"ea2b7bc6409a9854c0d0c0622c522c8aff575640853b79cc2bafb41993ae0399"} Sep 19 12:58:46.731785 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.731750 2585 scope.go:117] "RemoveContainer" containerID="472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55" Sep 19 12:58:46.743330 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.743309 2585 scope.go:117] "RemoveContainer" containerID="472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55" Sep 19 12:58:46.743817 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:58:46.743775 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55\": container with ID starting with 472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55 not found: ID does not exist" containerID="472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55" Sep 19 12:58:46.743931 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.743817 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55"} err="failed to get container status \"472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55\": rpc error: code = NotFound desc = could not find container \"472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55\": container with ID starting with 472dfa0e635f9024b48c14b0a48628a5ddf5e71f5762d511ff8f96f05226ee55 not found: ID does not exist" Sep 19 12:58:46.873756 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.873712 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-maas-api-tls\") pod \"33efb618-140c-42f1-a75c-708eb2de3bab\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " Sep 19 12:58:46.873942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.873775 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-metrics-certs\") pod \"33efb618-140c-42f1-a75c-708eb2de3bab\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " Sep 19 12:58:46.873942 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.873838 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7zlp\" (UniqueName: \"kubernetes.io/projected/33efb618-140c-42f1-a75c-708eb2de3bab-kube-api-access-n7zlp\") pod \"33efb618-140c-42f1-a75c-708eb2de3bab\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " Sep 19 12:58:46.874144 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.874099 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33efb618-140c-42f1-a75c-708eb2de3bab-openshift-service-ca\") pod \"33efb618-140c-42f1-a75c-708eb2de3bab\" (UID: \"33efb618-140c-42f1-a75c-708eb2de3bab\") " Sep 19 12:58:46.874451 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.874404 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33efb618-140c-42f1-a75c-708eb2de3bab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33efb618-140c-42f1-a75c-708eb2de3bab" (UID: "33efb618-140c-42f1-a75c-708eb2de3bab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:58:46.874984 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.874959 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33efb618-140c-42f1-a75c-708eb2de3bab-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:58:46.876143 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.876119 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33efb618-140c-42f1-a75c-708eb2de3bab" (UID: "33efb618-140c-42f1-a75c-708eb2de3bab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:58:46.876571 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.876554 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33efb618-140c-42f1-a75c-708eb2de3bab" (UID: "33efb618-140c-42f1-a75c-708eb2de3bab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:58:46.876648 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.876572 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33efb618-140c-42f1-a75c-708eb2de3bab-kube-api-access-n7zlp" (OuterVolumeSpecName: "kube-api-access-n7zlp") pod "33efb618-140c-42f1-a75c-708eb2de3bab" (UID: "33efb618-140c-42f1-a75c-708eb2de3bab"). InnerVolumeSpecName "kube-api-access-n7zlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:58:46.976187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.976111 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:58:46.976187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.976139 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33efb618-140c-42f1-a75c-708eb2de3bab-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:58:46.976187 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:46.976150 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zlp\" (UniqueName: \"kubernetes.io/projected/33efb618-140c-42f1-a75c-708eb2de3bab-kube-api-access-n7zlp\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:58:47.055573 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:47.055543 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4"] Sep 19 12:58:47.063804 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:47.063773 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1593f0-798bcc7449-dgnp4"] Sep 19 12:58:48.580741 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:58:48.580702 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33efb618-140c-42f1-a75c-708eb2de3bab" path="/var/lib/kubelet/pods/33efb618-140c-42f1-a75c-708eb2de3bab/volumes" Sep 19 12:59:12.509835 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.509795 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499"] Sep 19 12:59:12.510499 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.510106 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" podUID="96fd1262-763b-45d2-b285-492aca342d42" containerName="maas-api" containerID="cri-o://aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23" gracePeriod=30 Sep 19 12:59:12.786448 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.786421 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:59:12.904439 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.904407 2585 generic.go:356] "Generic (PLEG): container finished" podID="96fd1262-763b-45d2-b285-492aca342d42" containerID="aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23" exitCode=0 Sep 19 12:59:12.904610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.904453 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" Sep 19 12:59:12.904610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.904487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" event={"ID":"96fd1262-763b-45d2-b285-492aca342d42","Type":"ContainerDied","Data":"aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23"} Sep 19 12:59:12.904610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.904548 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499" event={"ID":"96fd1262-763b-45d2-b285-492aca342d42","Type":"ContainerDied","Data":"045817651a5baa165934ae5b07d384e7f12b92b1c03d822504f627330cfbb01c"} Sep 19 12:59:12.904610 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.904578 2585 scope.go:117] "RemoveContainer" containerID="aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23" Sep 19 12:59:12.914260 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.914238 2585 scope.go:117] "RemoveContainer" containerID="aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23" Sep 19 12:59:12.914584 ip-10-0-131-149 kubenswrapper[2585]: E0919 12:59:12.914560 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23\": container with ID starting with aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23 not found: ID does not exist" containerID="aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23" Sep 19 12:59:12.914724 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.914598 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23"} err="failed to get container status \"aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23\": rpc error: code = NotFound desc = could not find container \"aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23\": container with ID starting with aa9f502623e24a018de98f7b89f59fff8c78d5051709a6b650e7ca26c37cdb23 not found: ID does not exist" Sep 19 12:59:12.946007 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.945975 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fd1262-763b-45d2-b285-492aca342d42-openshift-service-ca\") pod \"96fd1262-763b-45d2-b285-492aca342d42\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " Sep 19 12:59:12.946141 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.946050 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-metrics-certs\") pod \"96fd1262-763b-45d2-b285-492aca342d42\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " Sep 19 12:59:12.946190 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.946171 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-maas-api-tls\") pod \"96fd1262-763b-45d2-b285-492aca342d42\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " Sep 19 12:59:12.946228 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.946209 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plqvv\" (UniqueName: \"kubernetes.io/projected/96fd1262-763b-45d2-b285-492aca342d42-kube-api-access-plqvv\") pod \"96fd1262-763b-45d2-b285-492aca342d42\" (UID: \"96fd1262-763b-45d2-b285-492aca342d42\") " Sep 19 12:59:12.946409 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.946389 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fd1262-763b-45d2-b285-492aca342d42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "96fd1262-763b-45d2-b285-492aca342d42" (UID: "96fd1262-763b-45d2-b285-492aca342d42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 12:59:12.946551 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.946531 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fd1262-763b-45d2-b285-492aca342d42-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:59:12.948481 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.948454 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "96fd1262-763b-45d2-b285-492aca342d42" (UID: "96fd1262-763b-45d2-b285-492aca342d42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:59:12.948567 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.948480 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "96fd1262-763b-45d2-b285-492aca342d42" (UID: "96fd1262-763b-45d2-b285-492aca342d42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 12:59:12.948567 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:12.948496 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fd1262-763b-45d2-b285-492aca342d42-kube-api-access-plqvv" (OuterVolumeSpecName: "kube-api-access-plqvv") pod "96fd1262-763b-45d2-b285-492aca342d42" (UID: "96fd1262-763b-45d2-b285-492aca342d42"). InnerVolumeSpecName "kube-api-access-plqvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 12:59:13.047603 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:13.047569 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:59:13.047603 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:13.047602 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fd1262-763b-45d2-b285-492aca342d42-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:59:13.047850 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:13.047612 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plqvv\" (UniqueName: \"kubernetes.io/projected/96fd1262-763b-45d2-b285-492aca342d42-kube-api-access-plqvv\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 12:59:13.229281 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:13.229251 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499"] Sep 19 12:59:13.237331 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:13.237297 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f32129-6bfb448d8c-4w499"] Sep 19 12:59:14.580046 ip-10-0-131-149 kubenswrapper[2585]: I0919 12:59:14.580009 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96fd1262-763b-45d2-b285-492aca342d42" path="/var/lib/kubelet/pods/96fd1262-763b-45d2-b285-492aca342d42/volumes" Sep 19 13:00:06.755230 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:06.755203 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 13:00:06.758828 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:06.758804 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 13:00:12.740750 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:12.740715 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5"] Sep 19 13:00:12.741338 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:12.741020 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" podUID="eff00aa4-b618-4325-af81-1400958a2faa" containerName="maas-api" containerID="cri-o://5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88" gracePeriod=30 Sep 19 13:00:12.997891 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:12.997821 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 13:00:13.179438 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.179396 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-metrics-certs\") pod \"eff00aa4-b618-4325-af81-1400958a2faa\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " Sep 19 13:00:13.179438 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.179454 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhj9b\" (UniqueName: \"kubernetes.io/projected/eff00aa4-b618-4325-af81-1400958a2faa-kube-api-access-xhj9b\") pod \"eff00aa4-b618-4325-af81-1400958a2faa\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " Sep 19 13:00:13.179761 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.179474 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls\") pod \"eff00aa4-b618-4325-af81-1400958a2faa\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " Sep 19 13:00:13.179761 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.179496 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff00aa4-b618-4325-af81-1400958a2faa-openshift-service-ca\") pod \"eff00aa4-b618-4325-af81-1400958a2faa\" (UID: \"eff00aa4-b618-4325-af81-1400958a2faa\") " Sep 19 13:00:13.180229 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.180195 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff00aa4-b618-4325-af81-1400958a2faa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eff00aa4-b618-4325-af81-1400958a2faa" (UID: "eff00aa4-b618-4325-af81-1400958a2faa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 13:00:13.182065 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.182031 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eff00aa4-b618-4325-af81-1400958a2faa" (UID: "eff00aa4-b618-4325-af81-1400958a2faa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:13.182440 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.182406 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eff00aa4-b618-4325-af81-1400958a2faa" (UID: "eff00aa4-b618-4325-af81-1400958a2faa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:13.182566 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.182541 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff00aa4-b618-4325-af81-1400958a2faa-kube-api-access-xhj9b" (OuterVolumeSpecName: "kube-api-access-xhj9b") pod "eff00aa4-b618-4325-af81-1400958a2faa" (UID: "eff00aa4-b618-4325-af81-1400958a2faa"). InnerVolumeSpecName "kube-api-access-xhj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:00:13.277169 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.277088 2585 generic.go:356] "Generic (PLEG): container finished" podID="eff00aa4-b618-4325-af81-1400958a2faa" containerID="5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88" exitCode=0 Sep 19 13:00:13.277169 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.277156 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" Sep 19 13:00:13.277353 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.277184 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" event={"ID":"eff00aa4-b618-4325-af81-1400958a2faa","Type":"ContainerDied","Data":"5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88"} Sep 19 13:00:13.277353 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.277281 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5" event={"ID":"eff00aa4-b618-4325-af81-1400958a2faa","Type":"ContainerDied","Data":"364fd14cb1109a84a84194a014656254cca28ee24640ec611405c6815d983319"} Sep 19 13:00:13.277353 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.277320 2585 scope.go:117] "RemoveContainer" containerID="5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88" Sep 19 13:00:13.280762 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.280738 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:13.280886 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.280772 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhj9b\" (UniqueName: \"kubernetes.io/projected/eff00aa4-b618-4325-af81-1400958a2faa-kube-api-access-xhj9b\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:13.280886 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.280817 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eff00aa4-b618-4325-af81-1400958a2faa-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:13.280886 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.280832 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff00aa4-b618-4325-af81-1400958a2faa-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:13.288672 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.288649 2585 scope.go:117] "RemoveContainer" containerID="5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88" Sep 19 13:00:13.288975 ip-10-0-131-149 kubenswrapper[2585]: E0919 13:00:13.288956 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88\": container with ID starting with 5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88 not found: ID does not exist" containerID="5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88" Sep 19 13:00:13.289045 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.288983 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88"} err="failed to get container status \"5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88\": rpc error: code = NotFound desc = could not find container \"5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88\": container with ID starting with 5fa3e1e1a6f58061d1d51aaec182f539e54e4d919fae386c64dca17ba6691a88 not found: ID does not exist" Sep 19 13:00:13.302559 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.302535 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5"] Sep 19 13:00:13.311892 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:13.311871 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4e754a-6856f454b7-pwgl5"] Sep 19 13:00:14.579027 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:14.578992 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff00aa4-b618-4325-af81-1400958a2faa" path="/var/lib/kubelet/pods/eff00aa4-b618-4325-af81-1400958a2faa/volumes" Sep 19 13:00:30.781316 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:30.780432 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc"] Sep 19 13:00:30.781316 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:30.780678 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" podUID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" containerName="maas-api" containerID="cri-o://fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8" gracePeriod=30 Sep 19 13:00:30.928700 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:30.928650 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" podUID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.85:8443/health\": dial tcp 10.132.0.85:8443: connect: connection refused" Sep 19 13:00:31.037574 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.037552 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 13:00:31.127246 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.127213 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prw2c\" (UniqueName: \"kubernetes.io/projected/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-kube-api-access-prw2c\") pod \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " Sep 19 13:00:31.127408 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.127258 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-metrics-certs\") pod \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " Sep 19 13:00:31.127408 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.127345 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls\") pod \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " Sep 19 13:00:31.127408 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.127366 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-openshift-service-ca\") pod \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\" (UID: \"0bbf5a75-b706-4958-a6c5-83aa91a96ed7\") " Sep 19 13:00:31.127843 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.127815 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0bbf5a75-b706-4958-a6c5-83aa91a96ed7" (UID: "0bbf5a75-b706-4958-a6c5-83aa91a96ed7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 13:00:31.129489 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.129457 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0bbf5a75-b706-4958-a6c5-83aa91a96ed7" (UID: "0bbf5a75-b706-4958-a6c5-83aa91a96ed7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:31.129599 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.129486 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0bbf5a75-b706-4958-a6c5-83aa91a96ed7" (UID: "0bbf5a75-b706-4958-a6c5-83aa91a96ed7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:31.129599 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.129509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-kube-api-access-prw2c" (OuterVolumeSpecName: "kube-api-access-prw2c") pod "0bbf5a75-b706-4958-a6c5-83aa91a96ed7" (UID: "0bbf5a75-b706-4958-a6c5-83aa91a96ed7"). InnerVolumeSpecName "kube-api-access-prw2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:00:31.228353 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.228317 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:31.228353 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.228346 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:31.228353 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.228356 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prw2c\" (UniqueName: \"kubernetes.io/projected/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-kube-api-access-prw2c\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:31.228580 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.228367 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbf5a75-b706-4958-a6c5-83aa91a96ed7-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:31.391104 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.391015 2585 generic.go:356] "Generic (PLEG): container finished" podID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" containerID="fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8" exitCode=0 Sep 19 13:00:31.391104 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.391070 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" Sep 19 13:00:31.391330 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.391107 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" event={"ID":"0bbf5a75-b706-4958-a6c5-83aa91a96ed7","Type":"ContainerDied","Data":"fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8"} Sep 19 13:00:31.391330 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.391158 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc" event={"ID":"0bbf5a75-b706-4958-a6c5-83aa91a96ed7","Type":"ContainerDied","Data":"ae26a42bf390a6468b751a5e3b2c5c8b3c08c983c3c57fa65df8ef868e4ab116"} Sep 19 13:00:31.391330 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.391182 2585 scope.go:117] "RemoveContainer" containerID="fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8" Sep 19 13:00:31.401820 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.401775 2585 scope.go:117] "RemoveContainer" containerID="fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8" Sep 19 13:00:31.402055 ip-10-0-131-149 kubenswrapper[2585]: E0919 13:00:31.402037 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8\": container with ID starting with fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8 not found: ID does not exist" containerID="fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8" Sep 19 13:00:31.402110 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.402062 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8"} err="failed to get container status \"fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8\": rpc error: code = NotFound desc = could not find container \"fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8\": container with ID starting with fd7bbc67f86ab77b363730f605f8b3996d448b2171bc54d5048434549a164fd8 not found: ID does not exist" Sep 19 13:00:31.413611 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.413587 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc"] Sep 19 13:00:31.420971 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:31.420943 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b0e30e-6978648bb4-h8tdc"] Sep 19 13:00:32.580386 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:32.580354 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" path="/var/lib/kubelet/pods/0bbf5a75-b706-4958-a6c5-83aa91a96ed7/volumes" Sep 19 13:00:48.759835 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:48.759753 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz"] Sep 19 13:00:48.760271 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:48.759964 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" podUID="472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" containerName="maas-api" containerID="cri-o://98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef" gracePeriod=30 Sep 19 13:00:49.021323 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.021263 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 13:00:49.085923 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.085884 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls\") pod \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " Sep 19 13:00:49.086143 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.085960 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l28v\" (UniqueName: \"kubernetes.io/projected/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-kube-api-access-8l28v\") pod \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " Sep 19 13:00:49.086471 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.086442 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-openshift-service-ca\") pod \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " Sep 19 13:00:49.086881 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.086857 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" (UID: "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 13:00:49.087043 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.087024 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-metrics-certs\") pod \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\" (UID: \"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75\") " Sep 19 13:00:49.087411 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.087391 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:49.088921 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.088880 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" (UID: "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:49.089054 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.089004 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-kube-api-access-8l28v" (OuterVolumeSpecName: "kube-api-access-8l28v") pod "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" (UID: "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75"). InnerVolumeSpecName "kube-api-access-8l28v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:00:49.089401 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.089376 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" (UID: "472b95cb-4cbd-4bc0-84a3-cb17e47c1d75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:49.188708 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.188653 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:49.188708 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.188709 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:49.188882 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.188726 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l28v\" (UniqueName: \"kubernetes.io/projected/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75-kube-api-access-8l28v\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:49.503748 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.503712 2585 generic.go:356] "Generic (PLEG): container finished" podID="472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" containerID="98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef" exitCode=0 Sep 19 13:00:49.503748 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.503742 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" Sep 19 13:00:49.504013 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.503795 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" event={"ID":"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75","Type":"ContainerDied","Data":"98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef"} Sep 19 13:00:49.504013 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.503858 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz" event={"ID":"472b95cb-4cbd-4bc0-84a3-cb17e47c1d75","Type":"ContainerDied","Data":"ea96e9319b854ca4fb7c3162a581b69d260b51b29192d95d3fd09bb859503161"} Sep 19 13:00:49.504013 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.503889 2585 scope.go:117] "RemoveContainer" containerID="98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef" Sep 19 13:00:49.514483 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.514467 2585 scope.go:117] "RemoveContainer" containerID="98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef" Sep 19 13:00:49.514785 ip-10-0-131-149 kubenswrapper[2585]: E0919 13:00:49.514762 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef\": container with ID starting with 98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef not found: ID does not exist" containerID="98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef" Sep 19 13:00:49.514848 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.514800 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef"} err="failed to get container status \"98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef\": rpc error: code = NotFound desc = could not find container \"98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef\": container with ID starting with 98d307eb0bcec87d15e8abac4fa9aa786f0707b49f988924d7053ce84caed9ef not found: ID does not exist" Sep 19 13:00:49.526660 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.526638 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz"] Sep 19 13:00:49.531874 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:49.531853 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d6124-565789f58-z7rbz"] Sep 19 13:00:50.578857 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:50.578819 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" path="/var/lib/kubelet/pods/472b95cb-4cbd-4bc0-84a3-cb17e47c1d75/volumes" Sep 19 13:00:58.159281 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.159250 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-rc65l"] Sep 19 13:00:58.159697 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.159436 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c69b6d687-rc65l" podUID="e1106c14-d2cb-4dce-8c8a-49c5187f0c56" containerName="manager" containerID="cri-o://f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb" gracePeriod=30 Sep 19 13:00:58.515032 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.515007 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 13:00:58.559813 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.559772 2585 generic.go:356] "Generic (PLEG): container finished" podID="e1106c14-d2cb-4dce-8c8a-49c5187f0c56" containerID="f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb" exitCode=0 Sep 19 13:00:58.559813 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.559812 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-rc65l" Sep 19 13:00:58.560039 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.559820 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-rc65l" event={"ID":"e1106c14-d2cb-4dce-8c8a-49c5187f0c56","Type":"ContainerDied","Data":"f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb"} Sep 19 13:00:58.560039 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.559920 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-rc65l" event={"ID":"e1106c14-d2cb-4dce-8c8a-49c5187f0c56","Type":"ContainerDied","Data":"a35fdade2d1b302506a3f7eff2eead2753e5548dfe814392663d2146969fb6c3"} Sep 19 13:00:58.560039 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.559953 2585 scope.go:117] "RemoveContainer" containerID="f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb" Sep 19 13:00:58.570995 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.570973 2585 scope.go:117] "RemoveContainer" containerID="f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb" Sep 19 13:00:58.571300 ip-10-0-131-149 kubenswrapper[2585]: E0919 13:00:58.571280 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb\": container with ID starting with f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb not found: ID does not exist" containerID="f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb" Sep 19 13:00:58.571410 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.571309 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb"} err="failed to get container status \"f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb\": rpc error: code = NotFound desc = could not find container \"f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb\": container with ID starting with f995a32adce9469070e6ec07e7c4780b20635e6679732c215c322c4f4c4719bb not found: ID does not exist" Sep 19 13:00:58.673725 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.673698 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-metrics-certs\") pod \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " Sep 19 13:00:58.673894 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.673776 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-webhook-cert\") pod \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " Sep 19 13:00:58.673894 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.673812 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr6gt\" (UniqueName: \"kubernetes.io/projected/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-kube-api-access-jr6gt\") pod \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\" (UID: \"e1106c14-d2cb-4dce-8c8a-49c5187f0c56\") " Sep 19 13:00:58.675975 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.675943 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e1106c14-d2cb-4dce-8c8a-49c5187f0c56" (UID: "e1106c14-d2cb-4dce-8c8a-49c5187f0c56"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:58.676078 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.676011 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e1106c14-d2cb-4dce-8c8a-49c5187f0c56" (UID: "e1106c14-d2cb-4dce-8c8a-49c5187f0c56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:00:58.676225 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.676203 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-kube-api-access-jr6gt" (OuterVolumeSpecName: "kube-api-access-jr6gt") pod "e1106c14-d2cb-4dce-8c8a-49c5187f0c56" (UID: "e1106c14-d2cb-4dce-8c8a-49c5187f0c56"). InnerVolumeSpecName "kube-api-access-jr6gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:00:58.774941 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.774912 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr6gt\" (UniqueName: \"kubernetes.io/projected/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-kube-api-access-jr6gt\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:58.774941 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.774938 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:58.775081 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.774950 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1106c14-d2cb-4dce-8c8a-49c5187f0c56-webhook-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:00:58.882861 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.882833 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-rc65l"] Sep 19 13:00:58.886271 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:00:58.886246 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-rc65l"] Sep 19 13:01:00.578605 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:00.578573 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1106c14-d2cb-4dce-8c8a-49c5187f0c56" path="/var/lib/kubelet/pods/e1106c14-d2cb-4dce-8c8a-49c5187f0c56/volumes" Sep 19 13:01:59.673874 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.673838 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c69b6d687-p4fwv"] Sep 19 13:01:59.674334 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674299 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33efb618-140c-42f1-a75c-708eb2de3bab" containerName="maas-api" Sep 19 13:01:59.674334 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674312 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="33efb618-140c-42f1-a75c-708eb2de3bab" containerName="maas-api" Sep 19 13:01:59.674334 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674325 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" containerName="maas-api" Sep 19 13:01:59.674334 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674330 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674341 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96fd1262-763b-45d2-b285-492aca342d42" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674346 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="96fd1262-763b-45d2-b285-492aca342d42" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674361 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674366 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674375 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff00aa4-b618-4325-af81-1400958a2faa" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674380 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff00aa4-b618-4325-af81-1400958a2faa" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674393 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1106c14-d2cb-4dce-8c8a-49c5187f0c56" containerName="manager" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674397 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1106c14-d2cb-4dce-8c8a-49c5187f0c56" containerName="manager" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674460 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="96fd1262-763b-45d2-b285-492aca342d42" containerName="maas-api" Sep 19 13:01:59.674465 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674467 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="33efb618-140c-42f1-a75c-708eb2de3bab" containerName="maas-api" Sep 19 13:01:59.674787 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674473 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1106c14-d2cb-4dce-8c8a-49c5187f0c56" containerName="manager" Sep 19 13:01:59.674787 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674478 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bbf5a75-b706-4958-a6c5-83aa91a96ed7" containerName="maas-api" Sep 19 13:01:59.674787 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674484 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff00aa4-b618-4325-af81-1400958a2faa" containerName="maas-api" Sep 19 13:01:59.674787 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.674489 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="472b95cb-4cbd-4bc0-84a3-cb17e47c1d75" containerName="maas-api" Sep 19 13:01:59.677408 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.677387 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.679645 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.679621 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 19 13:01:59.680580 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.680562 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 19 13:01:59.680732 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.680678 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhllr\"" Sep 19 13:01:59.680732 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.680698 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 19 13:01:59.688073 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.688051 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-p4fwv"] Sep 19 13:01:59.770524 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.770494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-webhook-cert\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.770726 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.770528 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9td\" (UniqueName: \"kubernetes.io/projected/f6dc7fdd-d98d-4173-a361-d82ea08d4999-kube-api-access-6x9td\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.770726 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.770559 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-metrics-certs\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.871416 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.871386 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-webhook-cert\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.871416 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.871417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9td\" (UniqueName: \"kubernetes.io/projected/f6dc7fdd-d98d-4173-a361-d82ea08d4999-kube-api-access-6x9td\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.871594 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.871442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-metrics-certs\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.873952 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.873925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-webhook-cert\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.874060 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.874038 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-metrics-certs\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.879566 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.879544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9td\" (UniqueName: \"kubernetes.io/projected/f6dc7fdd-d98d-4173-a361-d82ea08d4999-kube-api-access-6x9td\") pod \"maas-controller-6c69b6d687-p4fwv\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:01:59.988151 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:01:59.988085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:02:00.259412 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:00.259386 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-p4fwv"] Sep 19 13:02:00.260896 ip-10-0-131-149 kubenswrapper[2585]: W0919 13:02:00.260873 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dc7fdd_d98d_4173_a361_d82ea08d4999.slice/crio-b98e6945eb1ce14c2d3613365ac8ebc1f3794996defbd037f6707bb4ba690f0f WatchSource:0}: Error finding container b98e6945eb1ce14c2d3613365ac8ebc1f3794996defbd037f6707bb4ba690f0f: Status 404 returned error can't find the container with id b98e6945eb1ce14c2d3613365ac8ebc1f3794996defbd037f6707bb4ba690f0f Sep 19 13:02:00.262240 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:00.262221 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 13:02:00.915525 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:00.915490 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" event={"ID":"f6dc7fdd-d98d-4173-a361-d82ea08d4999","Type":"ContainerStarted","Data":"b98e6945eb1ce14c2d3613365ac8ebc1f3794996defbd037f6707bb4ba690f0f"} Sep 19 13:02:01.923215 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:01.923181 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" event={"ID":"f6dc7fdd-d98d-4173-a361-d82ea08d4999","Type":"ContainerStarted","Data":"d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f"} Sep 19 13:02:01.923585 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:01.923325 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:02:01.942859 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:01.942816 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" podStartSLOduration=1.9434648079999999 podStartE2EDuration="2.94280044s" totalImagesPullingTime="999.335632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 13:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 13:02:01.940897257 +0000 UTC m=+2217.547747841" watchObservedRunningTime="2026-09-19 13:02:01.94280044 +0000 UTC m=+2217.549651036" Sep 19 13:02:12.930873 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:12.930845 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:02:20.147076 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.146992 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-p4fwv"] Sep 19 13:02:20.147526 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.147231 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" podUID="f6dc7fdd-d98d-4173-a361-d82ea08d4999" containerName="manager" containerID="cri-o://d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f" gracePeriod=30 Sep 19 13:02:20.392374 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.392351 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:02:20.543616 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.543586 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-webhook-cert\") pod \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " Sep 19 13:02:20.543814 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.543650 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-metrics-certs\") pod \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " Sep 19 13:02:20.543814 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.543731 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x9td\" (UniqueName: \"kubernetes.io/projected/f6dc7fdd-d98d-4173-a361-d82ea08d4999-kube-api-access-6x9td\") pod \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\" (UID: \"f6dc7fdd-d98d-4173-a361-d82ea08d4999\") " Sep 19 13:02:20.545794 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.545765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6dc7fdd-d98d-4173-a361-d82ea08d4999" (UID: "f6dc7fdd-d98d-4173-a361-d82ea08d4999"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:02:20.545906 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.545811 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f6dc7fdd-d98d-4173-a361-d82ea08d4999" (UID: "f6dc7fdd-d98d-4173-a361-d82ea08d4999"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:02:20.545906 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.545827 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dc7fdd-d98d-4173-a361-d82ea08d4999-kube-api-access-6x9td" (OuterVolumeSpecName: "kube-api-access-6x9td") pod "f6dc7fdd-d98d-4173-a361-d82ea08d4999" (UID: "f6dc7fdd-d98d-4173-a361-d82ea08d4999"). InnerVolumeSpecName "kube-api-access-6x9td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:02:20.645419 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.645387 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-webhook-cert\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:02:20.645419 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.645416 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dc7fdd-d98d-4173-a361-d82ea08d4999-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:02:20.645572 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:20.645426 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9td\" (UniqueName: \"kubernetes.io/projected/f6dc7fdd-d98d-4173-a361-d82ea08d4999-kube-api-access-6x9td\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:02:21.039986 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.039954 2585 generic.go:356] "Generic (PLEG): container finished" podID="f6dc7fdd-d98d-4173-a361-d82ea08d4999" containerID="d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f" exitCode=0 Sep 19 13:02:21.040149 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.039996 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" Sep 19 13:02:21.040149 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.040041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" event={"ID":"f6dc7fdd-d98d-4173-a361-d82ea08d4999","Type":"ContainerDied","Data":"d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f"} Sep 19 13:02:21.040149 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.040088 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-p4fwv" event={"ID":"f6dc7fdd-d98d-4173-a361-d82ea08d4999","Type":"ContainerDied","Data":"b98e6945eb1ce14c2d3613365ac8ebc1f3794996defbd037f6707bb4ba690f0f"} Sep 19 13:02:21.040149 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.040108 2585 scope.go:117] "RemoveContainer" containerID="d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f" Sep 19 13:02:21.049216 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.049197 2585 scope.go:117] "RemoveContainer" containerID="d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f" Sep 19 13:02:21.049469 ip-10-0-131-149 kubenswrapper[2585]: E0919 13:02:21.049449 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f\": container with ID starting with d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f not found: ID does not exist" containerID="d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f" Sep 19 13:02:21.049546 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.049478 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f"} err="failed to get container status \"d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f\": rpc error: code = NotFound desc = could not find container \"d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f\": container with ID starting with d334005bf99e2d89cbf1768f3c0b5607ae1838d2ce3f7e79d42f8759bb6b9b1f not found: ID does not exist" Sep 19 13:02:21.070428 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.070404 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-p4fwv"] Sep 19 13:02:21.082179 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.082155 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-p4fwv"] Sep 19 13:02:21.850742 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.850677 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c69b6d687-2mxxc"] Sep 19 13:02:21.851330 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.851308 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6dc7fdd-d98d-4173-a361-d82ea08d4999" containerName="manager" Sep 19 13:02:21.851424 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.851332 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6dc7fdd-d98d-4173-a361-d82ea08d4999" containerName="manager" Sep 19 13:02:21.851481 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.851470 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6dc7fdd-d98d-4173-a361-d82ea08d4999" containerName="manager" Sep 19 13:02:21.855991 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.855969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:21.859143 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.859112 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhllr\"" Sep 19 13:02:21.859143 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.859112 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 19 13:02:21.859308 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.859162 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 19 13:02:21.859308 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.859181 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 19 13:02:21.865119 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.865097 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-2mxxc"] Sep 19 13:02:21.956056 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.956019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-metrics-certs\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:21.956056 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.956075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-webhook-cert\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:21.956243 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:21.956100 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2xs\" (UniqueName: \"kubernetes.io/projected/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-kube-api-access-nd2xs\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.057473 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.057445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-metrics-certs\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.057605 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.057484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-webhook-cert\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.057647 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.057610 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2xs\" (UniqueName: \"kubernetes.io/projected/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-kube-api-access-nd2xs\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.060115 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.060096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-metrics-certs\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.060226 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.060157 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-webhook-cert\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.065273 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.065253 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2xs\" (UniqueName: \"kubernetes.io/projected/c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4-kube-api-access-nd2xs\") pod \"maas-controller-6c69b6d687-2mxxc\" (UID: \"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4\") " pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.168058 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.167968 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:22.441705 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.441667 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c69b6d687-2mxxc"] Sep 19 13:02:22.443456 ip-10-0-131-149 kubenswrapper[2585]: W0919 13:02:22.443423 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2536c2c_1a5f_4dd2_a2ca_8987cbe822a4.slice/crio-61c9a96dfe2ae4bc85896f7a7d2e8630ad44eace493a5060244dac3e7aacc902 WatchSource:0}: Error finding container 61c9a96dfe2ae4bc85896f7a7d2e8630ad44eace493a5060244dac3e7aacc902: Status 404 returned error can't find the container with id 61c9a96dfe2ae4bc85896f7a7d2e8630ad44eace493a5060244dac3e7aacc902 Sep 19 13:02:22.580748 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:22.580720 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6dc7fdd-d98d-4173-a361-d82ea08d4999" path="/var/lib/kubelet/pods/f6dc7fdd-d98d-4173-a361-d82ea08d4999/volumes" Sep 19 13:02:23.063775 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:23.063731 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" event={"ID":"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4","Type":"ContainerStarted","Data":"175a81dff51829e252556854fcd141a0e64fa2b120a5f87692e014331985a084"} Sep 19 13:02:23.064482 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:23.063818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" event={"ID":"c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4","Type":"ContainerStarted","Data":"61c9a96dfe2ae4bc85896f7a7d2e8630ad44eace493a5060244dac3e7aacc902"} Sep 19 13:02:23.064482 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:23.063875 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:02:23.083021 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:23.082975 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" podStartSLOduration=1.750680776 podStartE2EDuration="2.082961006s" totalImagesPullingTime="332.28023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 13:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 13:02:23.080886889 +0000 UTC m=+2238.687737478" watchObservedRunningTime="2026-09-19 13:02:23.082961006 +0000 UTC m=+2238.689811587" Sep 19 13:02:34.072824 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:02:34.072792 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c69b6d687-2mxxc" Sep 19 13:03:44.265547 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:44.265511 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54"] Sep 19 13:03:44.266140 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:44.265781 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" containerID="cri-o://c41845aafc9465ee5b916f803c287ac03ec734fdc4df6461af707d552f915823" gracePeriod=10 Sep 19 13:03:45.566746 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:45.566707 2585 generic.go:356] "Generic (PLEG): container finished" podID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerID="c41845aafc9465ee5b916f803c287ac03ec734fdc4df6461af707d552f915823" exitCode=0 Sep 19 13:03:45.567113 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:45.566785 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" event={"ID":"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4","Type":"ContainerDied","Data":"c41845aafc9465ee5b916f803c287ac03ec734fdc4df6461af707d552f915823"} Sep 19 13:03:45.567113 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:45.566884 2585 scope.go:117] "RemoveContainer" containerID="ca0f381632d0760bf1e40a0de38b0f421fe4b2252316a1e93e722df1967ec161" Sep 19 13:03:46.011782 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.011756 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 13:03:46.075917 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.075885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-extensions-socket-volume\") pod \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " Sep 19 13:03:46.076090 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.075942 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6p9v\" (UniqueName: \"kubernetes.io/projected/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-kube-api-access-k6p9v\") pod \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\" (UID: \"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4\") " Sep 19 13:03:46.076258 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.076233 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" (UID: "2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 13:03:46.078095 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.078072 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-kube-api-access-k6p9v" (OuterVolumeSpecName: "kube-api-access-k6p9v") pod "2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" (UID: "2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4"). InnerVolumeSpecName "kube-api-access-k6p9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:03:46.177101 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.177032 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-extensions-socket-volume\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:03:46.177101 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.177057 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6p9v\" (UniqueName: \"kubernetes.io/projected/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4-kube-api-access-k6p9v\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:03:46.575967 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.575942 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" Sep 19 13:03:46.581352 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.581329 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54" event={"ID":"2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4","Type":"ContainerDied","Data":"00b04aa46f6c1e569229589bda3f94fe8ba7e11996eac215200c3c5acb0e9fa3"} Sep 19 13:03:46.581480 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.581406 2585 scope.go:117] "RemoveContainer" containerID="c41845aafc9465ee5b916f803c287ac03ec734fdc4df6461af707d552f915823" Sep 19 13:03:46.603051 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.603024 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54"] Sep 19 13:03:46.608934 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:46.608905 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dhq54"] Sep 19 13:03:48.579504 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:03:48.579465 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" path="/var/lib/kubelet/pods/2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4/volumes" Sep 19 13:04:44.963551 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.963516 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68"] Sep 19 13:04:44.964033 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.963988 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" Sep 19 13:04:44.964033 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.964001 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" Sep 19 13:04:44.964033 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.964017 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" Sep 19 13:04:44.964033 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.964023 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" Sep 19 13:04:44.964169 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.964104 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" Sep 19 13:04:44.964169 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.964112 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff3c7b8-2bfc-43df-97f9-3eff10ca07b4" containerName="manager" Sep 19 13:04:44.967200 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.967183 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:44.969718 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.969701 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tc7bx\"" Sep 19 13:04:44.980004 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:44.979983 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68"] Sep 19 13:04:45.019773 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.019743 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8230236b-7947-4d13-b6d9-0bdc6a5b93da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m8g68\" (UID: \"8230236b-7947-4d13-b6d9-0bdc6a5b93da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.019893 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.019854 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xp8\" (UniqueName: \"kubernetes.io/projected/8230236b-7947-4d13-b6d9-0bdc6a5b93da-kube-api-access-k8xp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m8g68\" (UID: \"8230236b-7947-4d13-b6d9-0bdc6a5b93da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.121046 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.121011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xp8\" (UniqueName: \"kubernetes.io/projected/8230236b-7947-4d13-b6d9-0bdc6a5b93da-kube-api-access-k8xp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m8g68\" (UID: \"8230236b-7947-4d13-b6d9-0bdc6a5b93da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.121202 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.121095 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8230236b-7947-4d13-b6d9-0bdc6a5b93da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m8g68\" (UID: \"8230236b-7947-4d13-b6d9-0bdc6a5b93da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.121458 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.121437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8230236b-7947-4d13-b6d9-0bdc6a5b93da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m8g68\" (UID: \"8230236b-7947-4d13-b6d9-0bdc6a5b93da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.129822 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.129803 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xp8\" (UniqueName: \"kubernetes.io/projected/8230236b-7947-4d13-b6d9-0bdc6a5b93da-kube-api-access-k8xp8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m8g68\" (UID: \"8230236b-7947-4d13-b6d9-0bdc6a5b93da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.277490 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.277397 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.582924 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.582897 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68"] Sep 19 13:04:45.585736 ip-10-0-131-149 kubenswrapper[2585]: W0919 13:04:45.585668 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8230236b_7947_4d13_b6d9_0bdc6a5b93da.slice/crio-1d5dccbbf65abd8249cf063b299fc6334683ac6151d81b9af5453734ba6d4e1b WatchSource:0}: Error finding container 1d5dccbbf65abd8249cf063b299fc6334683ac6151d81b9af5453734ba6d4e1b: Status 404 returned error can't find the container with id 1d5dccbbf65abd8249cf063b299fc6334683ac6151d81b9af5453734ba6d4e1b Sep 19 13:04:45.928429 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.928341 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" event={"ID":"8230236b-7947-4d13-b6d9-0bdc6a5b93da","Type":"ContainerStarted","Data":"026081caf30160dbf4fbd24666522f6b4a13760d1837947d62a652fb3c95b7b3"} Sep 19 13:04:45.928429 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.928392 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" event={"ID":"8230236b-7947-4d13-b6d9-0bdc6a5b93da","Type":"ContainerStarted","Data":"1d5dccbbf65abd8249cf063b299fc6334683ac6151d81b9af5453734ba6d4e1b"} Sep 19 13:04:45.928429 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.928421 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:04:45.951188 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:45.951142 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" podStartSLOduration=1.95112719 podStartE2EDuration="1.95112719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 13:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 13:04:45.948187105 +0000 UTC m=+2381.555037714" watchObservedRunningTime="2026-09-19 13:04:45.95112719 +0000 UTC m=+2381.557977769" Sep 19 13:04:56.938446 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:04:56.938415 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m8g68" Sep 19 13:05:06.798065 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:05:06.798040 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 13:05:06.802040 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:05:06.802011 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 13:06:34.764203 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.764167 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr"] Sep 19 13:06:34.767721 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.767677 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:34.772112 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.772078 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr888\"" Sep 19 13:06:34.779215 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.779182 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr"] Sep 19 13:06:34.915374 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.915341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98400518-ee9f-457e-bfb8-4a9bc263113e-maas-api-tls\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:34.915562 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.915381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98400518-ee9f-457e-bfb8-4a9bc263113e-openshift-service-ca\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:34.915562 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.915399 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98400518-ee9f-457e-bfb8-4a9bc263113e-metrics-certs\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:34.915562 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:34.915486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tf6z\" (UniqueName: \"kubernetes.io/projected/98400518-ee9f-457e-bfb8-4a9bc263113e-kube-api-access-5tf6z\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.016621 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.016515 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98400518-ee9f-457e-bfb8-4a9bc263113e-maas-api-tls\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.016621 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.016561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98400518-ee9f-457e-bfb8-4a9bc263113e-openshift-service-ca\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.016621 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.016585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98400518-ee9f-457e-bfb8-4a9bc263113e-metrics-certs\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.016946 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.016650 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tf6z\" (UniqueName: \"kubernetes.io/projected/98400518-ee9f-457e-bfb8-4a9bc263113e-kube-api-access-5tf6z\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.017637 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.017612 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98400518-ee9f-457e-bfb8-4a9bc263113e-openshift-service-ca\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.019429 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.019406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98400518-ee9f-457e-bfb8-4a9bc263113e-metrics-certs\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.019702 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.019667 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98400518-ee9f-457e-bfb8-4a9bc263113e-maas-api-tls\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.024466 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.024442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tf6z\" (UniqueName: \"kubernetes.io/projected/98400518-ee9f-457e-bfb8-4a9bc263113e-kube-api-access-5tf6z\") pod \"maas-api-68c86dfd56-m6qfr\" (UID: \"98400518-ee9f-457e-bfb8-4a9bc263113e\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.088526 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.088484 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:35.368397 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.368357 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr"] Sep 19 13:06:35.370803 ip-10-0-131-149 kubenswrapper[2585]: W0919 13:06:35.370768 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98400518_ee9f_457e_bfb8_4a9bc263113e.slice/crio-610af802efe3c9a017b5d3dd2925f69256c95f4a599cc16e15eff6c5dc3c3147 WatchSource:0}: Error finding container 610af802efe3c9a017b5d3dd2925f69256c95f4a599cc16e15eff6c5dc3c3147: Status 404 returned error can't find the container with id 610af802efe3c9a017b5d3dd2925f69256c95f4a599cc16e15eff6c5dc3c3147 Sep 19 13:06:35.605242 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:35.605205 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" event={"ID":"98400518-ee9f-457e-bfb8-4a9bc263113e","Type":"ContainerStarted","Data":"610af802efe3c9a017b5d3dd2925f69256c95f4a599cc16e15eff6c5dc3c3147"} Sep 19 13:06:36.613618 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:36.613577 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" event={"ID":"98400518-ee9f-457e-bfb8-4a9bc263113e","Type":"ContainerStarted","Data":"ece34d4659d67a8da3f4dff3f565fee50cf616ffeeed07e16b1ec76a2be740b2"} Sep 19 13:06:36.614106 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:36.613770 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:36.633642 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:36.633590 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" podStartSLOduration=2.221727874 podStartE2EDuration="2.633574923s" totalImagesPullingTime="411.847049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 13:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 13:06:36.631422151 +0000 UTC m=+2492.238272763" watchObservedRunningTime="2026-09-19 13:06:36.633574923 +0000 UTC m=+2492.240425502" Sep 19 13:06:42.625285 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:42.625258 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-m6qfr" Sep 19 13:06:42.674387 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:42.674350 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5"] Sep 19 13:06:42.674576 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:42.674551 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" podUID="a67b123f-64f5-40c1-920c-2bdea84b5b7c" containerName="maas-api" containerID="cri-o://b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9" gracePeriod=30 Sep 19 13:06:43.930281 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:43.930256 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 13:06:44.104933 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.104902 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67b123f-64f5-40c1-920c-2bdea84b5b7c-openshift-service-ca\") pod \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " Sep 19 13:06:44.105105 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.104987 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz9qz\" (UniqueName: \"kubernetes.io/projected/a67b123f-64f5-40c1-920c-2bdea84b5b7c-kube-api-access-kz9qz\") pod \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " Sep 19 13:06:44.105182 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.105107 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-maas-api-tls\") pod \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " Sep 19 13:06:44.105182 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.105152 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-metrics-certs\") pod \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\" (UID: \"a67b123f-64f5-40c1-920c-2bdea84b5b7c\") " Sep 19 13:06:44.105326 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.105303 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67b123f-64f5-40c1-920c-2bdea84b5b7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a67b123f-64f5-40c1-920c-2bdea84b5b7c" (UID: "a67b123f-64f5-40c1-920c-2bdea84b5b7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 13:06:44.105549 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.105532 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a67b123f-64f5-40c1-920c-2bdea84b5b7c-openshift-service-ca\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:06:44.107243 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.107215 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a67b123f-64f5-40c1-920c-2bdea84b5b7c" (UID: "a67b123f-64f5-40c1-920c-2bdea84b5b7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:06:44.107333 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.107257 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a67b123f-64f5-40c1-920c-2bdea84b5b7c" (UID: "a67b123f-64f5-40c1-920c-2bdea84b5b7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 13:06:44.107333 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.107324 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67b123f-64f5-40c1-920c-2bdea84b5b7c-kube-api-access-kz9qz" (OuterVolumeSpecName: "kube-api-access-kz9qz") pod "a67b123f-64f5-40c1-920c-2bdea84b5b7c" (UID: "a67b123f-64f5-40c1-920c-2bdea84b5b7c"). InnerVolumeSpecName "kube-api-access-kz9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 13:06:44.206418 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.206399 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz9qz\" (UniqueName: \"kubernetes.io/projected/a67b123f-64f5-40c1-920c-2bdea84b5b7c-kube-api-access-kz9qz\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:06:44.206418 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.206419 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-maas-api-tls\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:06:44.206560 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.206429 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a67b123f-64f5-40c1-920c-2bdea84b5b7c-metrics-certs\") on node \"ip-10-0-131-149.ec2.internal\" DevicePath \"\"" Sep 19 13:06:44.663859 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.663824 2585 generic.go:356] "Generic (PLEG): container finished" podID="a67b123f-64f5-40c1-920c-2bdea84b5b7c" containerID="b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9" exitCode=0 Sep 19 13:06:44.663859 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.663855 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" Sep 19 13:06:44.664092 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.663898 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" event={"ID":"a67b123f-64f5-40c1-920c-2bdea84b5b7c","Type":"ContainerDied","Data":"b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9"} Sep 19 13:06:44.664092 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.663965 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5" event={"ID":"a67b123f-64f5-40c1-920c-2bdea84b5b7c","Type":"ContainerDied","Data":"552486d00c0c41f2a7f1007e58f4f0d7a0702f324fd9413a9f8309b93f1a4bd7"} Sep 19 13:06:44.664092 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.663996 2585 scope.go:117] "RemoveContainer" containerID="b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9" Sep 19 13:06:44.675044 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.675022 2585 scope.go:117] "RemoveContainer" containerID="b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9" Sep 19 13:06:44.675327 ip-10-0-131-149 kubenswrapper[2585]: E0919 13:06:44.675306 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9\": container with ID starting with b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9 not found: ID does not exist" containerID="b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9" Sep 19 13:06:44.675427 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.675332 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9"} err="failed to get container status \"b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9\": rpc error: code = NotFound desc = could not find container \"b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9\": container with ID starting with b46e20beead5a3a04c58c0d45b0474cd012bcadcdd076db588bbcbe0340306a9 not found: ID does not exist" Sep 19 13:06:44.681148 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.681127 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5"] Sep 19 13:06:44.684608 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:44.684590 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-vw4v5"] Sep 19 13:06:46.591112 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:06:46.591026 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67b123f-64f5-40c1-920c-2bdea84b5b7c" path="/var/lib/kubelet/pods/a67b123f-64f5-40c1-920c-2bdea84b5b7c/volumes" Sep 19 13:07:10.762994 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:10.762962 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r8qf5_cc4281c1-7e85-41d6-8da0-45f3889bf2d8/manager/0.log" Sep 19 13:07:10.875143 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:10.875114 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lfshr_49eba3e5-c592-4646-9ad9-4b444d8de961/manager/0.log" Sep 19 13:07:10.996143 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:10.996118 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c69b6d687-2mxxc_c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4/manager/0.log" Sep 19 13:07:11.107607 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:11.107578 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pxnnj_870d3f82-f018-4b52-a1f1-4ac4b798469d/server/0.log" Sep 19 13:07:11.226560 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:11.226529 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4fpnf_7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98/manager/0.log" Sep 19 13:07:11.340853 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:11.340825 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774bc4964b-8mzpj_1b9f2b63-5f0d-4034-8134-5e5e2edcc518/manager/0.log" Sep 19 13:07:11.346167 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:11.346149 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774bc4964b-8mzpj_1b9f2b63-5f0d-4034-8134-5e5e2edcc518/copy-manifests/0.log" Sep 19 13:07:12.066296 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:12.066267 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68c86dfd56-m6qfr_98400518-ee9f-457e-bfb8-4a9bc263113e/maas-api/0.log" Sep 19 13:07:12.397388 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:12.397310 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lzkxs_4ae9128a-239b-465d-a853-b6adcd87fab3/postgres/0.log" Sep 19 13:07:13.254015 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:13.253983 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bt64x_63b51a2b-429f-46e4-a243-6451235cf14a/manager/0.log" Sep 19 13:07:13.362582 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:13.362559 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-tbg7q_d1edbb71-d2c8-4484-a07d-eef385de2370/manager/0.log" Sep 19 13:07:13.595162 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:13.595135 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-m8g68_8230236b-7947-4d13-b6d9-0bdc6a5b93da/manager/0.log" Sep 19 13:07:13.841156 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:13.841129 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-8vx4q_5234da79-d945-482d-84c2-b8393a4e1223/manager/0.log" Sep 19 13:07:14.436515 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:14.436481 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c9bc57f57-2m99s_6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a/kube-auth-proxy/0.log" Sep 19 13:07:14.771292 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:14.771219 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dx6xx_72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1/payload-pre-processing/0.log" Sep 19 13:07:14.995887 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:14.995858 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfdf79c4b-4q7kb_cb7ad890-3bbd-408c-b9f2-8975c8dd5542/router/0.log" Sep 19 13:07:15.435456 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:15.435428 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-rjpkz_b1006afd-75ae-40bd-bde6-04a50948a763/main/0.log" Sep 19 13:07:21.773581 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.773548 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rrtv4/must-gather-7f24b"] Sep 19 13:07:21.774004 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.773991 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67b123f-64f5-40c1-920c-2bdea84b5b7c" containerName="maas-api" Sep 19 13:07:21.774004 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.774005 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67b123f-64f5-40c1-920c-2bdea84b5b7c" containerName="maas-api" Sep 19 13:07:21.774106 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.774097 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67b123f-64f5-40c1-920c-2bdea84b5b7c" containerName="maas-api" Sep 19 13:07:21.778817 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.778800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:21.782597 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.782572 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rrtv4\"/\"default-dockercfg-xbfxk\"" Sep 19 13:07:21.782747 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.782610 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rrtv4\"/\"kube-root-ca.crt\"" Sep 19 13:07:21.782747 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.782617 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rrtv4\"/\"openshift-service-ca.crt\"" Sep 19 13:07:21.791168 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.791147 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rrtv4/must-gather-7f24b"] Sep 19 13:07:21.850012 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.849981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2mk\" (UniqueName: \"kubernetes.io/projected/274a78b9-c6ac-4902-8d64-624ded45d8c3-kube-api-access-rl2mk\") pod \"must-gather-7f24b\" (UID: \"274a78b9-c6ac-4902-8d64-624ded45d8c3\") " pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:21.850137 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.850115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/274a78b9-c6ac-4902-8d64-624ded45d8c3-must-gather-output\") pod \"must-gather-7f24b\" (UID: \"274a78b9-c6ac-4902-8d64-624ded45d8c3\") " pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:21.951215 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.951188 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2mk\" (UniqueName: \"kubernetes.io/projected/274a78b9-c6ac-4902-8d64-624ded45d8c3-kube-api-access-rl2mk\") pod \"must-gather-7f24b\" (UID: \"274a78b9-c6ac-4902-8d64-624ded45d8c3\") " pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:21.951376 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.951342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/274a78b9-c6ac-4902-8d64-624ded45d8c3-must-gather-output\") pod \"must-gather-7f24b\" (UID: \"274a78b9-c6ac-4902-8d64-624ded45d8c3\") " pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:21.951598 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.951578 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/274a78b9-c6ac-4902-8d64-624ded45d8c3-must-gather-output\") pod \"must-gather-7f24b\" (UID: \"274a78b9-c6ac-4902-8d64-624ded45d8c3\") " pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:21.960880 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:21.960857 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2mk\" (UniqueName: \"kubernetes.io/projected/274a78b9-c6ac-4902-8d64-624ded45d8c3-kube-api-access-rl2mk\") pod \"must-gather-7f24b\" (UID: \"274a78b9-c6ac-4902-8d64-624ded45d8c3\") " pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:22.088757 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:22.088735 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rrtv4/must-gather-7f24b" Sep 19 13:07:22.360539 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:22.360507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rrtv4/must-gather-7f24b"] Sep 19 13:07:22.361876 ip-10-0-131-149 kubenswrapper[2585]: W0919 13:07:22.361838 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274a78b9_c6ac_4902_8d64_624ded45d8c3.slice/crio-10b09c0afbc0511413b12949f216b67afff8ca835d5391375bf1524814cd0b43 WatchSource:0}: Error finding container 10b09c0afbc0511413b12949f216b67afff8ca835d5391375bf1524814cd0b43: Status 404 returned error can't find the container with id 10b09c0afbc0511413b12949f216b67afff8ca835d5391375bf1524814cd0b43 Sep 19 13:07:22.363622 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:22.363608 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 13:07:22.893967 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:22.893937 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrtv4/must-gather-7f24b" event={"ID":"274a78b9-c6ac-4902-8d64-624ded45d8c3","Type":"ContainerStarted","Data":"10b09c0afbc0511413b12949f216b67afff8ca835d5391375bf1524814cd0b43"} Sep 19 13:07:23.904417 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:23.904374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrtv4/must-gather-7f24b" event={"ID":"274a78b9-c6ac-4902-8d64-624ded45d8c3","Type":"ContainerStarted","Data":"3cc93cdba8518ebb9d70c1f59877e852ee269b1ccce517043e705bc2340aa54f"} Sep 19 13:07:23.904942 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:23.904431 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrtv4/must-gather-7f24b" event={"ID":"274a78b9-c6ac-4902-8d64-624ded45d8c3","Type":"ContainerStarted","Data":"f692d1bdb22bb56ffd5d4fddd759aa435cbd458a2a264515e36ee9a8451b1b8c"} Sep 19 13:07:23.927991 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:23.927937 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rrtv4/must-gather-7f24b" podStartSLOduration=1.869668626 podStartE2EDuration="2.927922928s" totalImagesPullingTime="1.058254302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 13:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 13:07:23.927172472 +0000 UTC m=+2539.534023052" watchObservedRunningTime="2026-09-19 13:07:23.927922928 +0000 UTC m=+2539.534773507" Sep 19 13:07:25.262805 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:25.262777 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9s4cm_ab7572c8-43f2-4f5d-9c64-fd450e74bd2c/global-pull-secret-syncer/0.log" Sep 19 13:07:25.449912 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:25.449886 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vl828_5502ec92-c349-4d1d-b324-c14f8e73841f/konnectivity-agent/0.log" Sep 19 13:07:25.475112 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:25.475085 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-149.ec2.internal_08bdc3a69a1fdcb162b43b9319667a66/haproxy/0.log" Sep 19 13:07:30.012751 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:30.012715 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bt64x_63b51a2b-429f-46e4-a243-6451235cf14a/manager/0.log" Sep 19 13:07:30.046910 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:30.046883 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-tbg7q_d1edbb71-d2c8-4484-a07d-eef385de2370/manager/0.log" Sep 19 13:07:30.123247 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:30.123219 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-m8g68_8230236b-7947-4d13-b6d9-0bdc6a5b93da/manager/0.log" Sep 19 13:07:30.224856 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:30.224796 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-8vx4q_5234da79-d945-482d-84c2-b8393a4e1223/manager/0.log" Sep 19 13:07:31.736819 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.736781 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/alertmanager/0.log" Sep 19 13:07:31.772395 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.772360 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/config-reloader/0.log" Sep 19 13:07:31.794136 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.794098 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/kube-rbac-proxy-web/0.log" Sep 19 13:07:31.815467 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.815442 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/kube-rbac-proxy/0.log" Sep 19 13:07:31.836288 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.836263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/kube-rbac-proxy-metric/0.log" Sep 19 13:07:31.856939 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.856908 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/prom-label-proxy/0.log" Sep 19 13:07:31.879783 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.879748 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_482bdbeb-d56a-44be-90e3-8b36f21b246a/init-config-reloader/0.log" Sep 19 13:07:31.921052 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.921010 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-bvjnf_22e8a67f-f965-40a9-a3eb-33cae5f93013/cluster-monitoring-operator/0.log" Sep 19 13:07:31.960478 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.960448 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6x9f9_02dae659-fd13-4d8c-ab31-e7663a42af3e/kube-state-metrics/0.log" Sep 19 13:07:31.980123 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:31.980095 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6x9f9_02dae659-fd13-4d8c-ab31-e7663a42af3e/kube-rbac-proxy-main/0.log" Sep 19 13:07:32.012610 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.012530 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6x9f9_02dae659-fd13-4d8c-ab31-e7663a42af3e/kube-rbac-proxy-self/0.log" Sep 19 13:07:32.136002 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.135976 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5w5c7_f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51/node-exporter/0.log" Sep 19 13:07:32.160371 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.160347 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5w5c7_f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51/kube-rbac-proxy/0.log" Sep 19 13:07:32.183895 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.183870 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5w5c7_f421ea6a-8ee3-4c1f-a3cd-a1b904d20f51/init-textfile/0.log" Sep 19 13:07:32.700475 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.700393 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-qvf4c_d44a0101-d10b-4f9a-b861-3de9ab9ef6dd/prometheus-operator-admission-webhook/0.log" Sep 19 13:07:32.736348 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.736300 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54fffcbc5-84dwj_c92e16cb-8664-43f5-8339-476b93ff0a82/telemeter-client/0.log" Sep 19 13:07:32.769327 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.769302 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54fffcbc5-84dwj_c92e16cb-8664-43f5-8339-476b93ff0a82/reload/0.log" Sep 19 13:07:32.804414 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.804361 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54fffcbc5-84dwj_c92e16cb-8664-43f5-8339-476b93ff0a82/kube-rbac-proxy/0.log" Sep 19 13:07:32.853547 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.853521 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d495d4cb-f69b5_bbcaf35f-2a69-4c0a-8b72-c0496aa00e80/thanos-query/0.log" Sep 19 13:07:32.883605 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.883578 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d495d4cb-f69b5_bbcaf35f-2a69-4c0a-8b72-c0496aa00e80/kube-rbac-proxy-web/0.log" Sep 19 13:07:32.922062 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.922032 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d495d4cb-f69b5_bbcaf35f-2a69-4c0a-8b72-c0496aa00e80/kube-rbac-proxy/0.log" Sep 19 13:07:32.948174 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.948127 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d495d4cb-f69b5_bbcaf35f-2a69-4c0a-8b72-c0496aa00e80/prom-label-proxy/0.log" Sep 19 13:07:32.978115 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:32.978038 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d495d4cb-f69b5_bbcaf35f-2a69-4c0a-8b72-c0496aa00e80/kube-rbac-proxy-rules/0.log" Sep 19 13:07:33.005958 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:33.005927 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57d495d4cb-f69b5_bbcaf35f-2a69-4c0a-8b72-c0496aa00e80/kube-rbac-proxy-metrics/0.log" Sep 19 13:07:34.417948 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:34.417919 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-5ckfm_1170f64c-37a6-414c-9206-7dd103f5692b/networking-console-plugin/0.log" Sep 19 13:07:35.031423 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.030336 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/2.log" Sep 19 13:07:35.035507 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.035473 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p7fnt_100ef318-668d-45f0-9162-2b73c34edfcc/console-operator/3.log" Sep 19 13:07:35.383257 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.383217 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv"] Sep 19 13:07:35.392388 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.392342 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.400136 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.400107 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv"] Sep 19 13:07:35.502338 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.502296 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-proc\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.503021 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.502990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/4e729e88-3993-453c-a2af-65e559bb9f96-kube-api-access-vq6cf\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.503176 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.503158 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-sys\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.503317 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.503298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-podres\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.503445 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.503430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-lib-modules\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.591200 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.591163 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7994d448f7-qb288_1065c848-981f-4604-a594-4fe04fd969d6/console/0.log" Sep 19 13:07:35.604585 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604553 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/4e729e88-3993-453c-a2af-65e559bb9f96-kube-api-access-vq6cf\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604749 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-sys\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604749 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-podres\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604749 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-lib-modules\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604749 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604713 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-sys\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604939 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-podres\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604939 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-lib-modules\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.604939 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604903 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-proc\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.605046 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.604981 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e729e88-3993-453c-a2af-65e559bb9f96-proc\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.613700 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.613656 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/4e729e88-3993-453c-a2af-65e559bb9f96-kube-api-access-vq6cf\") pod \"perf-node-gather-daemonset-fhdwv\" (UID: \"4e729e88-3993-453c-a2af-65e559bb9f96\") " pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:35.713458 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:35.713363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:36.018046 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:36.018002 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv"] Sep 19 13:07:36.020092 ip-10-0-131-149 kubenswrapper[2585]: W0919 13:07:36.020017 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e729e88_3993_453c_a2af_65e559bb9f96.slice/crio-e39794b3b5e699f0c9873dafec9aa21f3115696442bf38e385a3202d205aab6c WatchSource:0}: Error finding container e39794b3b5e699f0c9873dafec9aa21f3115696442bf38e385a3202d205aab6c: Status 404 returned error can't find the container with id e39794b3b5e699f0c9873dafec9aa21f3115696442bf38e385a3202d205aab6c Sep 19 13:07:36.312998 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:36.312970 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-s94vw_071e3fee-b7f4-4cd1-912a-7ef92690a98c/volume-data-source-validator/0.log" Sep 19 13:07:37.048298 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:37.048239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" event={"ID":"4e729e88-3993-453c-a2af-65e559bb9f96","Type":"ContainerStarted","Data":"0e3851d934698589590d30766b81eb9f10b6919dfd39b87727e390f3c4909f25"} Sep 19 13:07:37.048799 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:37.048313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" event={"ID":"4e729e88-3993-453c-a2af-65e559bb9f96","Type":"ContainerStarted","Data":"e39794b3b5e699f0c9873dafec9aa21f3115696442bf38e385a3202d205aab6c"} Sep 19 13:07:37.048799 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:37.048425 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:37.520959 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:37.520880 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gxnxk_e1aac6fa-0d58-43a6-bc0d-b2b46877c014/dns/0.log" Sep 19 13:07:37.542519 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:37.542493 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gxnxk_e1aac6fa-0d58-43a6-bc0d-b2b46877c014/kube-rbac-proxy/0.log" Sep 19 13:07:37.625407 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:37.625373 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2nn4d_d62b4d7a-0062-4806-ba22-6fc3dc29e3d2/dns-node-resolver/0.log" Sep 19 13:07:38.237039 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:38.237013 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5b7xp_7d3187f2-8b12-441d-bc4f-17bcaa9b12e9/node-ca/0.log" Sep 19 13:07:40.056486 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:40.056450 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c9bc57f57-2m99s_6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a/kube-auth-proxy/0.log" Sep 19 13:07:40.150054 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:40.150024 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dx6xx_72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1/payload-pre-processing/0.log" Sep 19 13:07:40.215303 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:40.215274 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfdf79c4b-4q7kb_cb7ad890-3bbd-408c-b9f2-8975c8dd5542/router/0.log" Sep 19 13:07:41.099731 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:41.099703 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2k8p6_0022f8b9-5089-446e-a7ef-bf5a9016d8e8/serve-healthcheck-canary/0.log" Sep 19 13:07:41.608233 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:41.608195 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-n6jfv_5eec9ccb-6d80-4c68-82d5-1cd49377dfdf/insights-operator/0.log" Sep 19 13:07:41.608809 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:41.608788 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-n6jfv_5eec9ccb-6d80-4c68-82d5-1cd49377dfdf/insights-operator/1.log" Sep 19 13:07:41.699629 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:41.699600 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7hjq_4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d/kube-rbac-proxy/0.log" Sep 19 13:07:41.721117 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:41.721093 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7hjq_4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d/exporter/0.log" Sep 19 13:07:41.741817 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:41.741780 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7hjq_4fe736dd-2ae1-4c4b-bd4e-a09b2fdfbb0d/extractor/0.log" Sep 19 13:07:43.070130 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:43.070099 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" Sep 19 13:07:43.090035 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:43.089978 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rrtv4/perf-node-gather-daemonset-fhdwv" podStartSLOduration=8.089959505 podStartE2EDuration="8.089959505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 13:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 13:07:37.083750618 +0000 UTC m=+2552.690601230" watchObservedRunningTime="2026-09-19 13:07:43.089959505 +0000 UTC m=+2558.696810086" Sep 19 13:07:44.093167 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.093140 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r8qf5_cc4281c1-7e85-41d6-8da0-45f3889bf2d8/manager/0.log" Sep 19 13:07:44.458516 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.458488 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lfshr_49eba3e5-c592-4646-9ad9-4b444d8de961/manager/0.log" Sep 19 13:07:44.522238 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.522203 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c69b6d687-2mxxc_c2536c2c-1a5f-4dd2-a2ca-8987cbe822a4/manager/0.log" Sep 19 13:07:44.543445 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.543409 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pxnnj_870d3f82-f018-4b52-a1f1-4ac4b798469d/server/0.log" Sep 19 13:07:44.661029 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.660988 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4fpnf_7ae87bc1-f5fa-4fd9-8bd7-f741f4450b98/manager/0.log" Sep 19 13:07:44.692997 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.692963 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774bc4964b-8mzpj_1b9f2b63-5f0d-4034-8134-5e5e2edcc518/manager/0.log" Sep 19 13:07:44.717565 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:44.717537 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774bc4964b-8mzpj_1b9f2b63-5f0d-4034-8134-5e5e2edcc518/copy-manifests/0.log" Sep 19 13:07:46.287884 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:46.287848 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zjpj2_ebb5ed03-5c19-404c-939e-0cd3732a1e55/openshift-lws-operator/0.log" Sep 19 13:07:46.617757 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:46.617071 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c9bc57f57-2m99s_6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a/kube-auth-proxy/0.log" Sep 19 13:07:46.692046 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:46.692016 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dx6xx_72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1/payload-pre-processing/0.log" Sep 19 13:07:46.745068 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:46.745011 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfdf79c4b-4q7kb_cb7ad890-3bbd-408c-b9f2-8975c8dd5542/router/0.log" Sep 19 13:07:49.121946 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:49.121912 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c9bc57f57-2m99s_6c6e34fa-2bd4-4fac-b7ba-3321b902cf5a/kube-auth-proxy/0.log" Sep 19 13:07:49.205258 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:49.205213 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dx6xx_72fa7a05-6fe2-41f6-acf2-54f1ac7b3bd1/payload-pre-processing/0.log" Sep 19 13:07:49.258919 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:49.258890 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfdf79c4b-4q7kb_cb7ad890-3bbd-408c-b9f2-8975c8dd5542/router/0.log" Sep 19 13:07:51.901082 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:51.901048 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-qrv2d_072899a6-72d1-489a-b564-01e58cb221ce/kube-storage-version-migrator-operator/1.log" Sep 19 13:07:51.901746 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:51.901729 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-qrv2d_072899a6-72d1-489a-b564-01e58cb221ce/kube-storage-version-migrator-operator/0.log" Sep 19 13:07:52.878018 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:52.877988 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8wc5j_a901150b-7247-4fb6-8577-e0bc8d519a2a/kube-multus/0.log" Sep 19 13:07:53.251638 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.251609 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/kube-multus-additional-cni-plugins/0.log" Sep 19 13:07:53.276720 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.276669 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/egress-router-binary-copy/0.log" Sep 19 13:07:53.313022 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.312990 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/cni-plugins/0.log" Sep 19 13:07:53.340376 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.340346 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/bond-cni-plugin/0.log" Sep 19 13:07:53.364346 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.364303 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/routeoverride-cni/0.log" Sep 19 13:07:53.387856 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.387830 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/whereabouts-cni-bincopy/0.log" Sep 19 13:07:53.409651 ip-10-0-131-149 kubenswrapper[2585]: I0919 13:07:53.409612 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj7nj_9691988f-8bb1-4962-9649-8baac63318bf/whereabouts-cni/0.log"